Systematic Review on Large Language Models in Orthopaedic Surgery

Area of Science:

• Orthopaedic Surgery
• Artificial Intelligence
• Medical Education

Background:

• The rapid development of Large Language Models (LLMs) since 2022 presents new opportunities in orthopaedic surgery.
• This systematic review is the first to examine the current research landscape of LLMs in the field.

Purpose of the Study:

• To identify LLMs researched in orthopaedics.
• To assess their functionalities and evaluate the quality of their results.
• To compare LLM performance against orthopaedic residents.

Main Methods:

• Systematic review conducted using PubMed, Embase, and Cochrane Library.
• Adherence to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.
• Inclusion of 60 studies evaluating LLMs like ChatGPT, Bard, Perplexity AI, and Bing.

Main Results:

• ChatGPT 4.0 demonstrated higher accuracy (47.2-73.6% without images) than ChatGPT 3.5 (29.4-55.8% without images).
• Bard achieved 49.8-58% accuracy; image-based assessments showed lower performance for all LLMs.
• Orthopaedic residents consistently outperformed LLMs, scoring 74.2-75.3%.

Conclusions:

• ChatGPT 4.0 significantly improved over ChatGPT 3.5 in orthopaedic assessment accuracy.
• Orthopaedic residents generally scored higher than current LLMs.
• Substantial opportunities exist for enhancing LLM performance in orthopaedic assessments, image analysis, and clinical documentation.