The Challenges of Using ChatGPT for Clinical Decision Support in Orthopaedic Surgery: A Pilot Study

Area of Science:

• Artificial Intelligence in Medicine
• Orthopaedic Surgery
• Clinical Decision Support Systems

Background:

• Artificial intelligence (AI) is increasingly accessible and applicable in healthcare.
• AI's capability for complex clinical decision-making, beyond simple queries, remains largely untested.
• This study investigates AI's potential for integrating diverse clinical data in orthopaedic surgery.

Purpose of the Study:

• To evaluate the capacity of AI, specifically ChatGPT4, to generate assessments and plans for orthopaedic surgery consultations.
• To determine if AI can synthesize multiple clinical data sources for complex patient cases.

Main Methods:

• Ten common fractures were selected from an academic trauma center's patient data (2022-2023).
• Patient consult notes (history, physical, imaging) were provided to ChatGPT4 in raw and semistructured formats.
• AI-generated assessments and plans were compared against the actual patient clinical outcomes determined by a trauma conference.

Main Results:

• ChatGPT4 produced safe and reasonable treatment plans that aligned with final patient outcomes.
• The AI demonstrated an ability to create effective plans using both raw and semistructured clinical data.
• Subjective comparison was necessary due to the lack of a standardized quantitative rubric for evaluating large language models.

Conclusions:

• ChatGPT can effectively synthesize complex clinical information (history, physical, imaging) into safe and reasonable orthopaedic surgery assessments and plans.
• AI shows promise in managing common fractures, indicating potential for broader clinical applications.
• Using actual clinical courses as benchmarks offers a viable research pathway for evaluating large language models in healthcare.