Related Experiment Videos
Evaluating artificial intelligence-generated clinical guidance for patellar dislocation: accuracy, readability and
Shuo Liu1,2, Xiyang Sun1,2, Yunfei Ma2
1Department of Orthopedics, Department of Sports Medicine and Arthroscopic Surgery, The First Affiliated Hospital of Anhui Medical University, Hefei, China.
Introduction:
Large language models (LLMs) are increasingly used to answer patient questions, but the readability and information quality of patellar dislocation guidance remain unclear.
Methods:
A series of clinical questions covering etiopathogenesis, mechanisms, clinical manifestations, diagnosis, treatment, complications, and postoperative rehabilitation of patellar dislocation were posed to DeepSeek-R1 v2.3.1 and ChatGPT-4o. Readability was assessed using the Flesch Reading Ease score (FRES), Flesch-Kincaid Grade Level (FKGL), Automated Readability Index (ARI), SMOG Index, and Gunning Fog Score (GFS). Three sports medicine surgeons rated responses using DISCERN. Wilcoxon signed-rank tests compared models; Inter-rater reliability (IRR) for DISCERN was assessed with intraclass correlation coefficient (ICC). Additionally, we analysed DeepSeek's responses based on the current literature and therapeutic standards.
Results:
Readability favored ChatGPT-4o across all indices (between-model P < 0.05): DeepSeek-R1 vs. ChatGPT-4o medians (Q1-Q3) were ARI 15.62 (13.90-16.60) vs. 12.54 (10.15-13.41), FKGL 14.39 (12.38-15.24) vs. 11.27 (9.44-12.76), GFS 18.25 (15.20-19.43) vs. 14.55 (13.73-16.53), SMOG 11.02 (9.72-12.18) vs. 9.55 (8.14-10.67), and FRES 9.50 (6.75-25.00) vs. 33.00 (19.75-41.50). Both models exceeded the NIH-recommended sixth-grade reading level (all P < 0.05). DISCERN scores were similar [median (Q1-Q3)] for DeepSeek-R1 vs. ChatGPT-4o: 49.17 (40.25-59.17) vs. 50.33 (38.92-58.42), corresponding to "Fair" quality (P = 0.767). DISCERN agreement was high (single-measure ICC 0.860-0.950).
Conclusions:
Although both AI platforms can generate clinically relevant responses of moderate quality, they should be regarded as adjunctive information rather than stand-alone clinical guidance, particularly given the high incidence of patellar instability and developmental characteristics of children and adolescents. Exploratory qualitative assessment of DeepSeek-R1 responses identified strengths in clinical explanation and reasoning, while also revealing limitations in citation reliability and readability. Future work should incorporate multi-turn, multi-generation frameworks, more samples with repeated sampling and longitudinal assessment, enforced citation standards, and expanded evaluation to better quantify both clinical utility and reliability.