Related Experiment Video
Updated: Sep 16, 2025
Quantitative Magnetic Resonance Imaging of Skeletal Muscle Disease
Published on: December 18, 2016
An Institutional Large Language Model for Musculoskeletal MRI Improves Protocol Adherence and Accuracy
James Thomas Patrick Decourcy Hallinan1,2, Naomi Wenxin Leow3, Yi Xian Low1
1Department of Diagnostic Imaging, National University Hospital, Singapore.
Background:
Privacy-preserving large language models (PP-LLMs) hold potential for assisting clinicians with documentation. We evaluated a PP-LLM to improve the clinical information on radiology request forms for musculoskeletal magnetic resonance imaging (MRI) and to automate protocoling, which ensures that the most appropriate imaging is performed.
Methods:
The present retrospective study included musculoskeletal MRI radiology request forms that had been randomly collected from June to December 2023. Studies without electronic medical record (EMR) entries were excluded. An institutional PP-LLM (Claude Sonnet 3.5) augmented the original radiology request forms by mining EMRs, and, in combination with rule-based processing of the LLM outputs, suggested appropriate protocols using institutional guidelines. Clinical information on the original and PP-LLM radiology request forms were compared with use of the RI-RADS (Reason for exam Imaging Reporting and Data System) grading by 2 musculoskeletal (MSK) radiologists independently (MSK1, with 13 years of experience, and MSK2, with 11 years of experience). These radiologists established a consensus reference standard for protocoling, against which the PP-LLM and of 2 second-year board-certified radiologists (RAD1 and RAD2) were compared. Inter-rater reliability was assessed with use of the Gwet AC1, and the percentage agreement with the reference standard was calculated.
Results:
Overall, 500 musculoskeletal MRI radiology request forms were analyzed for 407 patients (202 women and 205 men with a mean age [and standard deviation] of 50.3 ± 19.5 years) across a range of anatomical regions, including the spine/pelvis (143 MRI scans; 28.6%), upper extremity (169 scans; 33.8%) and lower extremity (188 scans; 37.6%). Two hundred and twenty-two (44.4%) of the 500 MRI scans required contrast. The clinical information provided in the PP-LLM-augmented radiology request forms was rated as superior to that in the original requests. Only 0.4% to 0.6% of PP-LLM radiology request forms were rated as limited/deficient, compared with 12.4% to 22.6% of the original requests (p < 0.001). Almost-perfect inter-rater reliability was observed for LLM-enhanced requests (AC1 = 0.99; 95% confidence interval [CI], 0.99 to 1.0), compared with substantial agreement for the original forms (AC1 = 0.62; 95% CI, 0.56 to 0.67). For protocoling, MSK1 and MSK2 showed almost-perfect agreement on the region/coverage (AC1 = 0.96; 95% CI, 0.95 to 0.98) and contrast requirement (AC1 = 0.98; 95% CI, 0.97 to 0.99). Compared with the consensus reference standard, protocoling accuracy for the PP-LLM was 95.8% (95% CI, 94.0% to 97.6%), which was significantly higher than that for both RAD1 (88.6%; 95% CI, 85.8% to 91.4%) and RAD2 (88.2%; 95% CI, 85.4% to 91.0%) (p < 0.001 for both).
Conclusions:
Musculoskeletal MRI request form augmentation with an institutional LLM provided superior clinical information and improved protocoling accuracy compared with clinician requests and non-MSK-trained radiologists. Institutional adoption of such LLMs could enhance the appropriateness of MRI utilization and patient care.
Level Of Evidence:
Diagnostic Level III . See Instructions for Authors for a complete description of levels of evidence.
Insights
Privacy-preserving large language models (PP-LLMs) significantly improved musculoskeletal MRI radiology request forms, enhancing clinical information and protocoling accuracy. This technology could optimize MRI utilization and patient care.
Area of Science:
- Medical Imaging
- Artificial Intelligence in Healthcare
- Radiology Informatics
Background:
- Privacy-preserving large language models (PP-LLMs) offer potential for enhancing clinical documentation.
- This study evaluated a PP-LLM to improve musculoskeletal MRI radiology request forms and automate protocoling.
Purpose of the Study:
- To assess the efficacy of a PP-LLM in augmenting clinical information on musculoskeletal MRI request forms.
- To evaluate the PP-LLM's ability to automate accurate radiology protocoling.
Main Methods:
- A retrospective analysis of 500 musculoskeletal MRI request forms was conducted.
- An institutional PP-LLM (Claude Sonnet 3.5) augmented forms by mining EMRs and suggesting protocols.
- Clinical information and protocoling accuracy were compared against a consensus reference standard established by expert radiologists.
Main Results:
- PP-LLM-augmented forms showed superior clinical information, with only 0.4%-0.6% rated deficient compared to 12.4%-22.6% for original forms (p < 0.001).
- Inter-rater reliability for LLM-enhanced requests was almost perfect (AC1 = 0.99) versus substantial for original forms (AC1 = 0.62).
- PP-LLM protocoling accuracy was 95.8%, significantly higher than human radiologists (88.6% and 88.2%, p < 0.001).
Conclusions:
- LLM augmentation of musculoskeletal MRI request forms yields superior clinical information and protocoling accuracy.
- Institutional adoption of PP-LLMs can improve MRI appropriateness and enhance patient care.
More Related Videos
08:19Protocol for the Evaluation of MRI Artifacts Caused by Metal Implants to Assess the Suitability of Implants and the Vulnerability of Pulse Sequences
Published on: May 17, 2018
08:39Longitudinal Micro-Computed Tomography Image Analysis for User-Defined Region of Interest in Critical-Sized Bone Defects
Published on: June 24, 2025
