An Evaluation of Interrater Reliability Measures on Binary Tasks Using d-Prime

Area of Science:

• Psychometrics
• Statistical Modeling
• Research Methodology

Background:

• Assessing interrater agreement is crucial for reliability in binary tasks.
• Existing indices for interrater agreement have faced criticism regarding their performance.
• A need exists for robust agreement indices that are insensitive to variations in prevalence, rater expertise, and bias.

Purpose of the Study:

• To evaluate the performance of five common interrater agreement indices on binary tasks.
• To identify the most reliable indices under varying conditions of characteristic prevalence, rater expertise, and bias.
• To provide evidence-based recommendations for selecting appropriate interrater agreement measures.

Main Methods:

• Monte Carlo simulations were employed to evaluate five interrater agreement indices.
• The performance of each index was assessed against d-prime, an unbiased measure of rater ability.
• Simulations varied characteristic prevalence, rater expertise, and rater bias to test index robustness.

Main Results:

• Phi and Kappa coefficients demonstrated superior performance across various simulation conditions.
• Percentage Agreement, Scott's Pi, and Gwet's AC1 showed lower correlations with d-prime.
• Phi and Kappa were least affected by changes in prevalence, expertise, and bias.

Conclusions:

• Phi and Kappa are recommended for assessing interrater agreement in binary tasks.
• These indices offer greater reliability and are less influenced by environmental and decision-maker characteristics.
• Researchers should prioritize Phi or Kappa for robust interrater reliability assessments.