Agreement Lambda for Weighted Disagreement With Ordinal Scales: Correction for Category Prevalence

Area of Science:

• Statistics
• Psychometrics
• Social Sciences

Background:

• Weighted inter-rater agreement is crucial for ordinal data.
• Existing methods (e.g., weighted Kappa) are sensitive to rater marginals and category prevalence.
• These methods often adjust for chance agreement, which may not be appropriate.

Purpose of the Study:

• Introduce a novel weighted Lambda coefficient for inter-rater agreement.
• Address limitations of existing weighted Kappa-like coefficients.
• Develop methods for statistical inference (standard errors, hypothesis tests, confidence intervals) for weighted Lambda.

Main Methods:

• Developed a new weighted Lambda coefficient that modifies observed agreement, incorporating prevalence-agreement effects.
• Proposed techniques for estimating sampling standard errors, hypothesis testing, and confidence intervals.
• Conducted Monte Carlo simulations and presented numerical examples to compare weighted Lambda with existing coefficients.

Main Results:

• Weighted Lambda effectively measures weighted inter-rater agreement, especially when accounting for prevalence-agreement effects.
• The new coefficient demonstrates advantages over traditional methods in various agreement scenarios.
• Simulations confirmed the utility and performance of weighted Lambda.

Conclusions:

• Weighted Lambda provides a robust alternative for measuring weighted inter-rater agreement.
• This coefficient offers improved handling of category prevalence and disagreement weighting.
• The proposed inferential methods facilitate the practical application of weighted Lambda.