Related Experiment Videos
An integrated deep learning and large language model for automated knee osteoarthritis grading and reporting from the
Dinh T Nguyen1,2, Duy K Hoang1,2, Linh N Luu2
1School of Biomedical Engineering, University of Technology Sydney, Sydney, Australia.
Objectives:
Knee osteoarthritis (KOA) is a prevalent joint condition affecting multiple structures of the knee. Clinicians need reliable tools for early detection and accurate grading to support timely management, but radiographic grading remains subjective and time-consuming. We aimed to develop a deep learning and large language model (LLM) hybrid system for KOA severity classification and automated clinical reporting.
Methods:
We analysed bilateral knee radiographs from 2435 adults (≥40 years) in the Vietnam Osteoporosis Study (VOS), using the Kellgren-Lawrence (KL) scale. Data were split into training (N = 1944) and independent testing (N = 491) sets. A hybrid deep learning pipeline combined knee-region object detection with KL grading, screening and feature localization. Outputs were integrated into a Gemini LLM module to generate structured clinical reports.
Results:
The hybrid model achieved 79.1% accuracy and AUC 88.8% for five-class KL grading, and 89.5% accuracy (sensitivity: 87.5%, specificity: 90.2%) for KOA screening. The localization module consistently highlighted KOA-relevant features, including osteophytes, joint space narrowing, and subchondral sclerosis. The Gemini module converted model outputs into coherent, structured reports and provided context-aware answers to clinician queries grounded in the imaging results.
Conclusions:
We developed a deployable deep learning-Gemini system for KOA assessment. This integrated framework offers an interpretable, automated approach to KOA assessment that may improve screening and decision support in resource-limited settings.
Demo:
https://www.youtube.com/watch?v=apXipdeXD5k.