Related Experiment Video
Updated: Oct 28, 2026
Screening of Axonal Degeneration in Carpal Tunnel Syndrome Using Ultrasonography and Nerve Conduction Studies
Published on: January 11, 2019
Evaluation of Large Language Model Concordance With AAOS Clinical Practice Guidelines for Carpal Tunnel Syndrome
Jonathan Spaan1, John T Streepy1, Alexander L Hornung1
1Rush University Medical Center, Chicago, IL, USA.
Background:
Large language models (LLMs) such as ChatGPT, Gemini, Claude, and Mistral are increasingly used to obtain medical information, but their accuracy in carpal tunnel syndrome (CTS) management is uncertain. This study evaluated whether LLM-generated recommendations align with American Academy of Orthopaedic Surgeons (AAOS) clinical practice guidelines (CPGs).
Methods:
Four LLMs (ChatGPT-4, Gemini, Claude-3, Mistral-7B) were queried with standardized prompts derived from AAOS CTS CPGs. Each prompt was asked 3 times per model, and responses were synthesized. Two blinded reviewers graded outputs as concordant, discordant, or indeterminate, with disagreements resolved by consensus. Concordance distributions were compared using χ2 testing with Bonferroni correction. Inter-rater reliability was measured with Cohen's κ.
Results:
Across 252 recommendations, 72.6% were concordant, 11.5% discordant, and 15.9% indeterminate. ChatGPT achieved the highest concordance (92.1%) and lowest discordance (1.6%), while Mistral performed poorest (52.4% concordance, 17.5% discordance). Reviewer agreement was excellent (κ = 0.91).
Conclusions:
Over 1 in 4 LLM-generated CTS recommendations were not concordant with AAOS CPGs. ChatGPT outperformed other models, though variable performance underscores the need for clinician oversight and careful validation before clinical use.
