Related Experiment Video
Updated: Jun 17, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Can Large Language Models (LLMs) Predict the Appropriate Treatment of Acute Hip Fractures in Older Adults? Comparing
Katrina S Nietsch1, Nancy Shrestha, Laura C Mazudie Ndjonko
1From the Icahn School of Medicine at Mount Sinai, New York, NY (Ms. Nietsch, Mr. Ahmed, Mr. Mejia, Mr. Zaidat, Ms. Ren, and Mr. Duey); the Chicago Medical School, Rosalind Franklin University of Medicine and Science, North Chicago, IL (Ms. Shrestha); the Northwestern University, Chicago, IL (Ms. Mazudie Ndjonko); the PGY-6, Department of Orthopedic Surgery and Neurosurgery, Mount Sinai Hospital, New York, NY (Dr. Li); the Department of Orthopedics and Orthopedic Surgery, Mount Sinai Hospital, New York, NY (Dr. Kim); the Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN (Dr. Hidden); and the Department of Orthopedic Surgery and Neurosurgery, Mount Sinai Hospital, New York, NY (Dr. Cho).
Background:
Acute hip fractures are a public health problem affecting primarily older adults. Chat Generative Pretrained Transformer may be useful in providing appropriate clinical recommendations for beneficial treatment.
Objective:
To evaluate the accuracy of Chat Generative Pretrained Transformer (ChatGPT)-4.0 by comparing its appropriateness scores for acute hip fractures with the American Academy of Orthopaedic Surgeons (AAOS) Appropriate Use Criteria given 30 patient scenarios. "Appropriateness" indicates the unexpected health benefits of treatment exceed the expected negative consequences by a wide margin.
Methods:
Using the AAOS Appropriate Use Criteria as the benchmark, numerical scores from 1 to 9 assessed appropriateness. For each patient scenario, ChatGPT-4.0 was asked to assign an appropriate score for six treatments to manage acute hip fractures.
Results:
Thirty patient scenarios were evaluated for 180 paired scores. Comparing ChatGPT-4.0 with AAOS scores, there was a positive correlation for multiple cannulated screw fixation, total hip arthroplasty, hemiarthroplasty, and long cephalomedullary nails. Statistically significant differences were observed only between scores for long cephalomedullary nails.
Conclusion:
ChatGPT-4.0 scores were not concordant with AAOS scores, overestimating the appropriateness of total hip arthroplasty, hemiarthroplasty, and long cephalomedullary nails, and underestimating the other three. ChatGPT-4.0 was inadequate in selecting an appropriate treatment deemed acceptable, most reasonable, and most likely to improve patient outcomes.
More Related Videos
09:51The Transition to an Anterior-Based Muscle Sparing Approach Improves Early Postoperative Function but is Associated with a Learning Curve
Published on: September 7, 2022
08:17A Mouse Model of Orthopedic Surgery to Study Postoperative Cognitive Dysfunction and Tissue Regeneration
Published on: February 27, 2018