Robustness of -type coefficients for clinical agreement

Area of Science:

• Statistics
• Psychometrics
• Data Analysis

Background:

• Inter-rater agreement is crucial for reliable data, often assessed using kappa-type coefficients.
• The behavior of these coefficients can be affected by asymmetric frequency distributions.
• Existing benchmarking methods may overlook experimental condition influences.

Purpose of the Study:

• To investigate the robustness of four kappa-type coefficients for nominal and ordinal data.
• To evaluate an inferential benchmarking procedure considering experimental conditions.
• To provide guidelines for designing robust inter-rater agreement studies.

Main Methods:

• Conducted an extensive Monte Carlo simulation study.
• Examined robustness across various scenarios: sample size, scale dimension, number of raters, and frequency distributions.
• Assessed four kappa-type coefficients and an inferential benchmarking procedure.

Main Results:

• Fleiss kappa and Conger kappa exhibited more paradoxical behavior with ordinal than nominal classifications.
• Coefficient robustness improved with increased sample size and number of raters for both classification types.
• Robustness improved with rating scale dimension only for nominal classifications.

Conclusions:

• Kappa-type coefficients show varying robustness depending on classification type and study design parameters.
• Larger sample sizes and more raters enhance the robustness of agreement coefficients.
• Guidelines are provided for optimal study design to ensure reliable inter-rater agreement assessment.