Investigating Expert-Rater Agreement and Inter/Intra-Rater Reliability of Two Fundamental Movement Skills for the...

The Furtado-Gallagher Children Observational Movement Pattern Assessment System (FG-COMPASS) is an observational tool using sequential decisions to assess fundamental movement skill proficiency. The current version of the test has three locomotor and five manipulative skills. Adding two more locomotor skills to the assessment tool enriches its scope, enabling a more comprehensive and nuanced evaluation of individual movement skills. We assessed expert-non-expert rater agreement and inter/intra non-expert rater reliability of two new scales for the locomotor subscale. We divided this study into two parts. In Part I, we filmed 60 children aged 5-10 years old who performed gallop and vertical jump skills. A motor behavior expert then classified the videotapes using our newly created rating scales. Next, we selected eight videos for training purposes and 24 videos for testing purposes. In Part II, 30 undergraduate students underwent rater training. Rating data were analyzed using weighted kappa (Kw) and the intra-class correlation coefficient (ICC), and these indices showed 'very good' agreement between the expert and the non-expert raters for vertical jump (Kw = .96) and gallop (Kw = .89). The ICC expert to non-expert rater values for vertical jump and gallop were .98 and .94, respectively; and mean kappa values for inter-rater reliability between non-experts were considered 'very good' for vertical jump (MKw = .92) and 'good' for gallop (MKw = .78). The ICC inter-rater values were .98 and .95 (considered 'excellent') for vertical jump and gallop, respectively; and the kappa intra-rater values were .96 and .85, respectively, with intra-rater ICC values .98 and .92. Thus, the proposed rating scales were reliable for assessing vertical jump and gallop. Future studies should focus on criterion-related validity and reliability evidence from live performances.