Examining the role of ChatGPT in the management of distal radius fractures: insights into its accuracy and consistency

Christopha J Knee ¹, Ryan J Campbell ¹, David J Graham ^2,3,4,5,6

¹Department of Orthopaedics and Trauma Surgery, Royal North Shore Hospital, St Leonards, New South Wales, Australia.
²Australian Research Collaboration on Hands [ARCH], Mudgeeraba, Queensland, Australia.
³School of Medicine and Dentistry, Griffith University, Southport, Queensland, Australia.
⁴School of Medicine, University of Queensland, Herston, Queensland, Australia.
⁵Department of Musculoskeletal Services, Gold Coast University Hospital, Southport, Queensland, Australia.
⁶Department of Orthopaedic Surgery, Queensland Children's Hospital, South Brisbane, Queensland, Australia.
⁷Department of Orthopaedic Surgery, St George Hospital, Kogarah, New South Wales, Australia.
⁸Department of Hand and Peripheral Nerve Surgery, Royal North Shore Hospital, St Leonards, New South Wales, Australia.
⁹Faculty of Medicine and Health, Sydney Medical School, The University of Sydney, Camperdown, New South Wales, Australia.
¹⁰Department of Orthopaedic Surgery, Hornsby Ku-ring-gai Hospital, Hornsby, New South Wales, Australia.
¹¹Department of Orthopaedic Surgery, Nepean Hospital, Kingswood, New South Wales, Australia.

⁰Department of Orthopaedics and Trauma Surgery, Royal North Shore Hospital, St Leonards, New South Wales, Australia.

Anz Journal of Surgery +

|

July 5, 2024

View abstract on PubMed

Summary

ChatGPT shows significant limitations in accuracy and consistency for managing distal radius fractures. Its current capabilities offer limited utility for patient education and clinical decision-making in orthopaedics.

Area Of Science

Orthopaedic Surgery
Artificial Intelligence in Medicine
Medical Information Systems

Background

Distal radius fractures present ongoing management challenges for orthopaedic surgeons.
Artificial Intelligence (AI) and Large Language Models (LLMs) show potential to enhance healthcare and research.
ChatGPT, a prominent LLM, is being explored for its application in medical contexts.

Purpose Of The Study

To evaluate the accuracy and consistency of ChatGPT's knowledge regarding distal radius fracture management.
To assess ChatGPT's suitability for patient education and orthopaedic clinical decision support.

Main Methods

ChatGPT was queried with seven questions on distal radius fracture management across two sessions, yielding 14 responses.
Responses covered patient inquiries and clinical decision-making scenarios.
Orthopaedic registrars and senior surgeons evaluated the accuracy, consistency, and reference validity of ChatGPT's outputs.

Main Results

All 14 responses contained a mixture of accurate and inaccurate information.
Of 47 cited references, only 13% were accurate; 28% were fabricated, and 57% were incorrect.
Response consistency was observed in 71% of cases, but overall accuracy was limited.

Conclusions

ChatGPT exhibits significant limitations in accuracy and consistency for information on distal radius fractures.
Current ChatGPT performance restricts its utility for patient education and clinical decision-making in this orthopaedic domain.

Keywords:

Artificial intelligence ChatGPT Distal radius fractures Generative pre‐trained transformer Large language models Patient education

Related Concept Videos

Fractures: Bone Repair

Treatment for a fracture is based on the type of break, the bone affected, and the patient's age.
Minor fractures with no bone displacement are treated by immobilizing the fractured bone using a cast or splint. However, in the case of fractures with displaced bones, the broken bones are repositioned before immobilization to ensure successful healing without deformation and loss of function. The realignment of fractured bone ends is performed through a process called reduction. If the...