Accounting for Patient Characteristics in a Model-Based Kappa of Agreement Between Two Experts' Ordinal Ratings

Cohen's kappa and other summary measures are often used in clinical studies to describe agreement and association between two experts' ordered categorical ratings. However, a key limitation of Cohen's kappa and similar measures is their inability to evaluate the impact of patient-related factors such as family history and age on the agreement and association between experts. Strong agreement between experts is an essential component of effective clinical procedures where subjective interpretation of patients' images or test results by an expert is required, for example, in the visual assessment of breast density from a mammogram. Not accounting for important patient-related factors can lead to inflated and biased assessments of agreement and association. In this article, our objective is to propose novel model-based measures that appropriately account for the impact of patient-related covariates on chance-corrected agreement and association between two experts' ordinal ratings that overcome limitations of existing measures. Our population-based approach is based on an ordinal generalized linear mixed model (GLMM). Rigorous simulation studies evaluating performance of the new model-based measures in a broad range of settings are reported. Existing and new measures are compared in two clinical applications assessing breast density and multiple sclerosis. Key advantages of the new kappa measures over existing measures such as Cohen's kappa include incorporating patient-related factors, robustness to underlying disease prevalence and marginal distributions of experts' ratings, and appropriately correcting for chance agreement. Sample R code is provided by the authors for application of proposed measures in other studies.