Related Experiment Video
Updated: Oct 28, 2026
Development of a Novel Internal Fixation Model for Rat Radial Fractures: Fracture Healing Assessment and Dorsal Root Ganglion Isolation
Published on: March 13, 2026
Performance of ChatGPT-5 under a minimal binary prompt for detection of distal radius fractures on plain radiographs
Bana Awad1,2, Yaniv Keren3, Dror Ronel3
1Technion - Israel Institute of Technology, The Ruth and Bruce Rappaport Faculty of Medicine, Haifa, 3109601, Israel. bana.awad@technion.ac.il.
Background:
General-purpose multimodal large language models (LLMs) are being applied to radiology, but their performance under the simplest realistic clinical query (a single binary fracture-or-not prompt) has not been rigorously quantified. We evaluated ChatGPT-5 on this task using a public distal radius dataset.
Methods:
In this retrospective diagnostic accuracy study, 396 anteroposterior distal radius radiographs from the Stanford MURA dataset were submitted to ChatGPT-5, each in a fresh temporary chat session. The prompt was: "Is there a fracture in this X-ray? Answer with yes or no." The reference standard was an image-level label assigned by consensus between a senior orthopedic resident and a board-certified orthopedic surgeon; MURA study-level labels were used only to select candidate images. Twenty-six radiographs (6.6%) returned a hedged response and were excluded from the primary analysis, leaving 370; all 26 were added back in three sensitivity analyses. Outcomes were sensitivity, specificity, PPV, NPV, F1, balanced accuracy, and Cohen's kappa, with 95% Wilson confidence intervals and McNemar's test for error asymmetry.
Results:
Fracture prevalence in the analyzed sample was 45.9% (170/370). Sensitivity was 51.2% (95% CI 43.7-58.6), specificity 70.5% (63.8-76.4), accuracy 61.6% (56.6-66.4), PPV 59.6% (51.5-67.2), NPV 62.9% (56.5-69.0), F1 0.551, Cohen's kappa 0.219 (0.112-0.318). False negatives exceeded false positives (McNemar χ2 = 4.06, p = 0.044). Non-binary responses occurred in 7.6% of fracture-positive and 5.7% of fracture-negative radiographs (p = 0.54); counting all as errors lowered sensitivity to 47.3% (40.2-54.5), and no assumption raised it above 54.9%. Projected to 20% prevalence, PPV was 30.3% and NPV 85.2%.
Conclusion:
Under a minimal binary prompt and without task-specific fine-tuning, ChatGPT-5 missed approximately half of distal radius fractures and fell well below published performance for dedicated CNN-based fracture detectors. Results are specific to this model version, this prompt, this anatomical region, and this dataset; they constrain claims about autonomous, minimal-prompt use rather than the broader potential of engineered LLM-based radiological workflows.
