Challenges and Misinterpretations of Cohen's Kappa in Agreement Studies in Ophthalmology

Cohen's kappa is widely used to measure rater agreement on binary outcomes and performs well when outcome prevalence is relatively balanced. This paper aims to explain the challenges encountered with Cohen's kappa, and to provide guidance to veterinary ophthalmologists on the usage and interpretation of kappa values.

Article based on an existing dataset in which two examiners graded iridocorneal angle abnormalities in 60 client-owned dogs during pre-breeding examinations.

Following the 2022 ECVO-HED (European College of Veterinary Ophthalmology-Hereditary Eye Diseases) gonioscopy grading scheme and its general recommendations, eyes were classified into two categories-"breeding-YES" (not affected, mildly or moderately affected) or "breeding-NO" (severely affected). Inter-examiner agreement was separately assessed for real (60 left and 58 right eyes) and simulated data (manually balanced left eye dataset with unchanged agreement between examiners).

While the two examiners classified 52/60 left and 51/58 right eyes into the "breeding-YES" category, they disagreed on 7/60 left and 1/58 right eyes, with resulting kappa values of 0.18 and 0.91 for the left and right eyes, respectively. After balancing the data, with both examiners classifying 27/60 left eyes into the "breeding-yes" category, but still disagreeing on 7/60 eyes, the kappa value increased to 0.77.

Readers should be cautious when comparing kappa values across studies with different underlying prevalence and bias. Under non-ideal data distribution, additional statistical indices like prevalence- and bias index, prevalence-and-bias-adjusted kappa (PABAK), and maxKappa, do not replace Cohen's kappa, but assist in interpreting it.