A comparison between over-the-shoulder and computer-derived measurement procedures in assessing student performance...

On July 18, 1980, the air traffic control Radar Training Facility (RTF) at Oklahoma City was formally dedicated. The RTF can use the computer system in making automated measures of how well a student is moving air traffic. The purpose of this study was to compare the computer-derived automated measures with over-the-shoulder evaluation methods. Subjects were 48 recent nonradar Federal Aviation Administration (FAA) Academy air traffic control graduates who were employed at the National Aviation Facilities Experimental Center (NAFEC; renamed FAA Technical Center) in Atlantic City, NJ, on the Dynamic Simulation Facility. Two sets of measures were taken: 20 computer-derived measures and an over-the-shoulder evaluation of student performance by experts in air traffic control. The results of the study show that the automated measures predict a success criterion at least as well as an over-the-shoulder evaluation and that the automated measures are far more reliable. The study further demonstrated that the factor structures of the two sets of measures were very similar except in two areas where direct observation was necessary for evaluation.