Inter-rater reliability of query/probe-based techniques for measuring situation awareness

Area of Science:

• Human Factors
• Cognitive Psychology
• Nuclear Engineering

Background:

• Situation awareness (SA) measures often use expert judgment for evaluation in real-world settings.
• This reliance on experts introduces variability, necessitating inter-rater reliability assessment.
• Existing literature inadequately addresses the inter-rater reliability of query-based SA measures.

Purpose of the Study:

• To investigate the inter-rater reliability of a query-based SA measure.
• To assess the impact of expert experience on SA measure reliability.
• To highlight the importance of expert judgment in SA measurement.

Main Methods:

• Recruited process experts to create reference keys for SA queries.
• Conducted trials using a full-scope nuclear power plant simulator.
• Analyzed agreement levels between expert raters for SA queries.

Main Results:

• The query-based SA measure exhibited only 'moderate' inter-rater reliability.
• Agreement levels varied significantly between expert pairs with differing experimental exposure.
• Directly phrased queries did not guarantee high inter-rater reliability.

Conclusions:

• Inter-rater reliability of query/probe-based SA measures should not be assumed in representative settings.
• Domain and experimental knowledge are critical for experts to provide reliable judgments.
• Practitioners must assess or ensure inter-rater reliability when using expert responses as ground truth for SA measures.