Related Experiment Video
Updated: Oct 24, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Accuracy of Large Language Model Chatbots in Retrieving MRI Safety Conditions for Medical Implants
Ariel Kerpel1, Shir Tal1, Mark Waserman1
1Department of Radiology, Chaim Sheba Medical Center, Tel Hashomer, Israel; Affiliated With the Gray Faculty of Medical and Health Sciences, Tel Aviv University, Tel Aviv, Israel.
Background:
Retrieving MRI safety conditions for implanted devices is increasingly complex and time-consuming.
Purpose:
To evaluate the accuracy of internet-enabled large language model (LLM) chatbots in retrieving implant-specific MRI safety conditions, using manufacturer documentation as the reference standard.
Study Type:
Prospective comparative evaluation in a constructed device cohort, followed by a retrospective clinical pilot.
Population:
Fifty purposively selected device configurations and 20 patients (10 with cardiac and 10 with noncardiac devices) referred to the MRI Safety Committee.
Field Strength:
No field-strength or sequence restrictions.
Assessment:
Standardized single-shot prompts requested eight parameters from ChatGPT-5.0, Gemini 2.5 Pro, and Claude Sonnet 4.5: MRI safety label, field strengths, eligible regions/centering, transmit-coil restrictions, RF exposure limits, maximum spatial gradient, maximum gradient slew rate, and scan duration. Two radiologists (10 and 25 years' experience) and an MRI technologist (32 years) independently scored 1200 parameter-level outputs against manufacturer Instructions for use and technical manuals using a 5-point scale (5, fully correct; 1, substantively incorrect or fabricated); scores of 4-5 indicated accuracy. In the pilot, 160 outputs from ChatGPT-5.6, a newer version of the best-performing chatbot, were scored identically.
Statistical Tests:
Intraclass correlation coefficient; Gwet's first-order agreement coefficient (AC1); Friedman test; Wilcoxon signed-rank tests; device-clustered bootstrap 95% confidence intervals (CIs); two-sided significance threshold, p < 0.05.
Results:
Mean ordinal scores were 4.76 (ChatGPT), 4.20 (Gemini), and 4.17 (Claude); ChatGPT scored significantly higher. Respective pooled parameter-level accuracies were 0.94 (95% CI, 0.89-0.98), 0.79 (0.71-0.86), and 0.78 (0.69-0.87). Pilot accuracy with ChatGPT-5.6 was 0.98 (95% CI, 0.94-1.00).
Data Conclusion:
Internet-enabled LLM chatbots correctly retrieved up to 94% of MRI safety parameters in the constructed cohort and 98% in the clinical pilot. Errors occurred even with the best-performing chatbot, requiring direct verification against current manufacturer documentation before clinical use.
Evidence Level:
Stage 2.
Technical Efficacy:
Stage 2.
