Accounting for Patient Characteristics in a Model-Based Kappa of Agreement Between Two Experts' Ordinal Ratings

Area of Science:

• Biostatistics
• Medical Imaging Analysis
• Clinical Trial Methodology

Background:

• Cohen's kappa is widely used for inter-rater reliability in clinical settings.
• Existing measures fail to account for patient-specific covariates influencing expert agreement.
• Ignoring patient factors can lead to biased assessments of agreement and association.

Purpose of the Study:

• To introduce novel model-based measures for assessing chance-corrected agreement and association between experts.
• To incorporate patient-related covariates into the evaluation of inter-rater reliability.
• To overcome limitations of existing measures like Cohen's kappa.

Main Methods:

• Development of population-based measures using an ordinal generalized linear mixed model (GLMM).
• Rigorous simulation studies to evaluate measure performance across diverse settings.
• Comparison of new and existing measures in clinical applications (breast density, multiple sclerosis).

Main Results:

• Proposed measures effectively incorporate patient-related factors into agreement assessment.
• New measures demonstrate robustness to variations in disease prevalence and rating distributions.
• Model-based approach provides appropriate correction for chance agreement.

Conclusions:

• Novel kappa measures offer improved accuracy by accounting for patient covariates.
• These measures enhance the reliability of expert assessments in clinical practice.
• The approach provides a more nuanced understanding of inter-rater agreement in medical contexts.