Large Language Model Use Cases in Health Care Research Are Redundant and Often Lack Appropriate Methodological...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Orthopaedic Surgery Research

Background:

• Large language models (LLMs) are increasingly explored for applications in healthcare.
• Musculoskeletal medicine presents numerous opportunities for LLM integration.
• Evaluating current LLM use and methodological rigor is crucial for future clinical implementation.

Purpose of the Study:

• To systematically review and categorize current use cases of LLMs in musculoskeletal medicine.
• To critically assess the methodological quality of studies investigating LLMs in orthopaedics.
• To identify areas for improvement in LLM research conduct and application.

Main Methods:

• Comprehensive literature search across PubMed, Cochrane Library, and Embase databases.
• Inclusion of studies evaluating LLMs in any orthopaedic surgery setting (clinical or educational).
• Methodological quality assessment using the Methodological Index for Non-Randomized Studies criteria.

Main Results:

• 114 studies (2022-2024) identified, revealing significant use case redundancy.
• Top applications include answering patient questions (42.1%), diagnosis/management (21.1%), and examination preparation (18.4%).
• Methodological inconsistencies were prevalent, with poor reporting of prompts (31.6%), outputs (56.1%), and limited bias investigation.

Conclusions:

• Current LLM research in musculoskeletal medicine is characterized by limited scope and redundant applications.
• Significant methodological inconsistencies, including lack of transparency in prompts and outputs, hinder reliable implementation.
• Urgent need for standardized guidance on methodological conduct, transparency, and novel use cases for LLMs in orthopaedics.