Accuracy Is Not Enough: Reasoning and Reference Reliability in Orthopaedic Large Language Model (LLM) Applications.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Orthopaedic Surgery Training

Background:

• Large language models (LLMs) demonstrate performance on par with postgraduate trainees in orthopaedics.
• Clinicians increasingly use LLMs for educational and decision-support purposes.
• Current LLM evaluations focus on accuracy, neglecting reasoning quality and reference reliability.

Purpose of the Study:

• To systematically evaluate the relationship between answer accuracy, reasoning quality, and reference reliability of LLMs on a postgraduate orthopaedic examination.
• To assess GPT-5's performance on the 2024 Orthopaedic In-Training Examination (OITE).

Main Methods:

• GPT-5 was administered the 2024 OITE (203 questions), providing answers, rationales, and references.
• Accuracy was compared against American Academy of Orthopaedic Surgeons (AAOS) data.
• A subsample underwent detailed validation of reasoning and references, comparing GPT-5's reasoning to AAOS explanations.

Main Results:

• GPT-5 achieved 78.3% accuracy, surpassing the OITE pass threshold and PGY-5 resident scores.
• Hallucinations occurred in 33% of responses, significantly higher in incorrect (50%) versus correct answers (15.9%).
• Reasoning for correct answers was high, with 95.5% matching AAOS explanations; however, 33% of all answers cited fabricated or misrepresented references.

Conclusions:

• GPT-5 demonstrates high accuracy on the OITE but exhibits poor reference reliability.
• Even accurate answers may depend on flawed or unverifiable sources.
• Evaluating LLMs in medical education requires assessing reasoning and evidence validation alongside accuracy.