Evaluating Large Language Models for Patient-Facing Total Knee Arthroplasty Information in Knee Osteoarthritis: Development and Application of the Composite Clinical Reliability Score
Related Experiment Videos
Sathish Muthu1, Dhibin Vikash Kolarpatti Ponnusamy2, Manimekalai Muthiah Rathinam3
1Department of Research Methods, Orthopaedic Research Group, Coimbatore, Tamil Nadu, India; Central Research Laboratory, Aarupadai Veedu Medical College and Hospital, Vinayaka Mission's Research Foundation (Deemed to be University), Puducherry, India.
Background:
Total knee arthroplasty (TKA) is the definitive surgical treatment for advanced knee osteoarthritis. As patients increasingly consult large language models (LLMs) for medical guidance, reliability, safety, and readability are concerns. This study compared patient-facing TKA information from three LLMs using the Composite Clinical Reliability Score (CCRS).
Methods:
We used 90 patient-facing TKA questions from Google's People Also Ask to test three LLMs. We employed two fellowship-trained orthopaedic surgeons to independently score the 270 responses at two timepoints across six CCRS domains: factual accuracy, United States Medical Licensing Examination (USMLE)-style clinical correctness, Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guideline adherence, hallucination severity index (HSI, inverse), clinical safety score (CSS), and evidence citation validity index (ECVI). Readability was assessed with the Coleman-Liau Index (CLI), corroborated by the Flesch-Kincaid Grade Level and Simple Measure of Gobbledygook (SMOG). Inter-rater reliability used the intraclass correlation coefficient (ICC), and inter-model differences the Friedman test.
Results:
Program 2 achieved the highest CCRS (median 84.0, interquartile range [IQR] 82.0 to 85.0), outperforming Program 3 (76.5, 65.0 to 83.8) and Program 1 (73.0, 69.0 to 79.0) (P < 0.0001), with superior accuracy, guideline adherence, hallucination control, and safety framing. Program 3 showed stronger ECVI (7.7 ± 1.5) than Program 2 (7.1 ± 0.8) and Program 1 (5.6 ± 0.6). Readability differed significantly (P < 0.0001): Program 2 was most accessible (CLI 12.3 ± 1.6), Program 1 intermediate (12.8 ± 1.7), and Program 3 least readable (13.7 ± 2.5). Agreement was excellent (ICC 0.87, 95% confidence interval [CI] 0.82 to 0.91).
Conclusion:
Within this framework and evaluation period, Program 2 ranked highest on composite reliability and readability, Program 3 offered better evidence traceability, but less accessible text, and Program 1 was intermediate. Because LLM outputs are non-deterministic and continuously updated, these findings are exploratory rather than a durable ranking, and LLMs should serve only as adjuncts to professional consultation.