Related Experiment Video
Updated: Jul 12, 2026
Minimally Invasive Treatment for Thoracolumbar Burst Fracture Using Sagittal Alignment Screws and A Trauma Reduction Device
Published on: November 8, 2024
Performance of DeepSeek-R1 and ChatGPT-5 in the Generation of North American Spine Society Clinical Guidelines for
Ruiyuan Chen1, Yue Pan1,2, Minghui Liang1
1Department of Orthopedics, Beijing Chao-Yang Hospital, Capital Medical University, 5 JingYuan Road, Shijingshan District, Beijing, 100043, China, 86 51718688.
Background:
Vertebral compression fractures (VCFs) impose a substantial clinical and health care burden, and their management relies on timely access to evidence-based guidelines. Large language models (LLMs) may help clinicians rapidly obtain guideline-related information, but their performance on VCF guidelines remains unclear.
Objective:
This study aimed to evaluate the performance of LLMs, including DeepSeek-R1 and ChatGPT-5, in generating responses consistent with VCF clinical guidelines.
Methods:
Using the 2024 North American Spine Society VCF clinical guidelines as the reference standard, 34 open-ended and 87 closed-ended questions were submitted to DeepSeek-R1 and ChatGPT-5. Four senior spine surgeons independently rated responses to both closed-ended and open-ended questions using a 5-point Likert scale for accuracy, consistency, self-awareness, and fabrication/falsification. For open-ended questions, comprehensiveness, clarity, and trust and confidence were additionally assessed. Subgroup analyses were performed by question type, recommendation grade, and VCF subtype, with direct comparisons between models.
Results:
A total of 726 responses were generated for 121 questions. For closed-ended questions, ChatGPT-5 and DeepSeek-R1 showed comparable performance in accuracy (P=.11), self-awareness (P=.10), and fabrication/falsification (P=.10). DeepSeek-R1 demonstrated better consistency than ChatGPT-5 for both closed-ended and open-ended questions (P<.001 and P=.001, respectively). For open-ended questions, the models differed significantly in comprehensiveness (P=.03) and trust and confidence (P=.02), but not in accuracy (P=.42), self-awareness (P=.22), fabrication/falsification (P=.64), or clarity (P=.48). Closed-ended questions generally outperformed open-ended questions. Responses to grade A-C recommendations outperformed grade I recommendations in accuracy, consistency, and fabrication/falsification (all P≤.001) but scored lower in self-awareness (P<.001). No significant differences were observed across VCF subtypes.
Conclusions:
Under a standardized clinician-oriented prompting condition, ChatGPT-5 and DeepSeek-R1 showed generally high but variable scores across evaluation dimensions, with important deficiencies remaining, particularly in interventional and surgical treatment recommendations and in questions linked to recommendation grade I. Because these findings were obtained in a controlled prompting setting, caution is warranted when extrapolating them to other query styles, clinical scenarios, or LLMs.
