Preliminary inter-rater reliability of "The observed Off-task Behavior among School-Children" (The OBS-Children)

This study investigated the inter-rater reliability of The observed Off-task Behavior among School Children (The OBS-Children), developed to measure observed off-task behavior among children from 6 to 12 years old with sensory processing difficulties and sensory seeking behavior. Two raters rated 18 videos of 10 children in a controlled school-like setting. Inter-rater reliability was calculated using Intraclass Correlation Coefficients (ICCs 2.1). Standard Error of Measurement (SEM) and Limits of Agreement (LoA) were calculated as a parameter of measurement error. Four out of six items had good to excellent reliability (ICC 0.83-0.95), while one had moderate reliability (ICC 0.75) and one had poor reliability (ICC 0.28). SEM was mostly small, while LoA ranged quite widely on all but one item. The reliability of the OBS-Children can be considered acceptable except from two problematic items. Measurement error was relatively large for all items, implying a need for further development and psychometric testing of the tool.