Development of a valid and reliable video-based decision-making test for Australian football umpires

To develop a valid and reliable video-based decision-making test to examine and monitor the decision-making performance of Australian football umpires.

Validation assessments with test re-test reliability.

A video-based decision-making test was developed from a pool of 156 video-based decision-making situations. Australian football umpires (n=56) and players (n=45) participated in two separate phases of analysis. In phase one, players completed a test re-test reliability assessment with a 100 video-clips. Results indicated 24 clips were a reliable measure of decision-making performance. In phase two, umpires completed a test re-test protocol with 80 clips, 24 of which were the reliable clips identified by the player cohort in phase one. These 24 clips provided a measure of construct validity. Face and content validity were assessed by skill acquisition specialists, expert umpire coaches, and umpires.

From each of phase one and two of the reliability assessment, 24 clips were found to have a kappa value greater than 0.30, providing a total of 48 reliable video-clips. Construct validity was supported, with the umpire group performing significantly better than the player group on the 24 clips presented to both groups on each testing occasion. Face and content validity were also demonstrated.

This investigation demonstrated the ability prospectively to determine reliability and validity of the video-based decision-making test designed specifically for Australian football umpires. Establishing the validity and reliability of the video clips ensures future investigations can accurately and consistently measure the decision-making performance of Australian football umpires.