From variability to consistency: building a Kellgren-Lawrence gonarthrosis dataset

This study aims to address variability in Kellgren-Lawrence (KL) staging by analyzing interobserver differences between radiologists and orthopedic surgeons to build unbiased datasets for AI-driven gonarthrosis solutions.

This retrospective study analyzed 15,159 weight-bearing AP knee radiographs from patients aged 18 years and older collected from five centers between 2011 and 2020. The radiographs were labeled by two radiologists and two orthopedic surgeons independently, followed by a consensus evaluation by two experienced orthopedic specialists. KL staging was assessed both on a 0-4 scale and in two clinically relevant categories: early stage (KL ≤ 2) and advanced stage (KL ≥ 3). In clinical practice, low stage cases are generally managed with conservative treatments, while high-stage cases often indicate a higher need for surgical intervention.

A total of 15,159 knee radiographs were analyzed, with participants comprising 6,207 (40.9%) males and 8,952 (59.1%) females, with a mean age of 56.97 ± 4.2 years. The agreement between radiologists and experts in the KL staging ranged from κ = 0.085 to κ = 0.172, while agreement between orthopedic surgeons and experts ranged from κ = 0.18 to κ = 0.307. These values indicate slight to fair agreement among radiologists and fair to moderate agreement among orthopedic surgeons on the original KL stage The categorization of the KL stages into two groups (KL ≤ 2 vs. KL ≥ 3) resulted in enhanced inter-rater reliability, with orthopedic surgeons demonstrating substantial agreement (κ = 0.637-0.692) compared to radiologists (κ = 0.183-0.230). The highest intra-professional agreement was observed between Radiologist 1 and Radiologist 2 (κ = 0.55) on the 0-4 the KL scale and between Orthopedic Surgeon 1 and Orthopedic Surgeon 2 (κ = 0.642) in the two-category classification.

Orthopedic surgeons showed higher agreement with experts and among themselves, particularly in advanced stages (KL ≥ 3), compared to radiologists, who demonstrated only mild to moderate agreement. Categorizing KL stages into early (KL ≤ 2) and advanced (KL ≥ 3) groups improved interobserver reliability, thereby facilitating the development of standardized AI datasets.