Not Interchangeable: Pilot Self-Reports and Instructor Competency Ratings as Cognitive-State Labels in Real Flight Operations

ObjectiveTo quantify the correspondence between pilots' post-task workload and situation-awareness ratings and instructor competency ratings during real flight.BackgroundAdaptive systems, neuroergonomic classifiers, and competency-based training all need cognitive-state labels. Self-reports capture felt demand and awareness; instructor evaluations capture behaviorally inferred competence.MethodForty pilots completed five Cessna 172 tasks during dedicated research flights. After each task, pilots gave single-item 0-10 ratings of workload and situation awareness. One flight instructor at each of the two participating organizations rated Situation Awareness and Workload Management on 1-4 International Civil Aviation Organization-aligned scales, and rated task execution with behavior-specific items, before hearing the self-ratings. Analyses used Spearman correlations with participant-clustered bootstrap intervals, within-person decomposition, and mixed-model and stratified sensitivity analyses.ResultsPooled cross-source correspondence was weak for situation awareness ($\rho =.08$) and workload ($\rho =-.14$; $.14$ aligned). The two instructor competencies were strongly intercorrelated ($\rho =.61$) and strongly associated with same-instructor task-execution ratings ($\rho =.69$ and $.72$). A small within-person workload signal ($r=-.19$) attenuated after double centering. Point estimates were similar in both strata and after excluding pilots aged 60 or older.ConclusionThe two sources provided distinct, non-substitutable labels. Their strong link to same-rater task-execution scores is consistent with execution-referenced judgment, though shared-rater variance cannot be ruled out; self-reports captured pilots' reported experience.ApplicationCognitive-state monitoring and training assessment should use multi-source labels and should not treat observation-based competency ratings as direct measures of internal workload or situation awareness.