Coefficient Lambda for Interrater Agreement Among Multiple Raters: Correction for Category Prevalence

Area of Science:

• Statistics
• Psychometrics
• Biostatistics

Background:

• Fleiss's Kappa extends Cohen's Kappa for multiple raters but has interpretative challenges.
• Assumptions of random chance agreement and rater number sensitivity limit its application.
• Category prevalence and characteristics influence agreement beyond random chance.

Purpose of the Study:

• To address limitations of Fleiss's Kappa.
• To introduce a novel agreement coefficient.
• To provide a more accurate interrater reliability measure for imbalanced category distributions.

Main Methods:

• Developed a new agreement coefficient adjusting for category prevalence.
• Shifted from random rater assignment assumption.
• Examined theoretical justification, interpretability, and standard error.

Main Results:

• The novel coefficient offers a more accurate measure of interrater reliability.
• It accounts for category prevalence, improving interpretation with imbalanced data.
• Simulations and practical applications demonstrate robustness.

Conclusions:

• The proposed coefficient enhances interrater reliability assessment.
• It provides a more nuanced understanding of agreement by considering category prevalence.
• This method offers improved interpretability and accuracy in statistical analysis.