An Institutional Large Language Model for Musculoskeletal MRI Improves Protocol Adherence and Accuracy

Area of Science:

• Medical Imaging
• Artificial Intelligence in Healthcare
• Radiology Informatics

Background:

• Privacy-preserving large language models (PP-LLMs) offer potential for enhancing clinical documentation.
• This study evaluated a PP-LLM to improve musculoskeletal MRI radiology request forms and automate protocoling.

Purpose of the Study:

• To assess the efficacy of a PP-LLM in augmenting clinical information on musculoskeletal MRI request forms.
• To evaluate the PP-LLM's ability to automate accurate radiology protocoling.

Main Methods:

• A retrospective analysis of 500 musculoskeletal MRI request forms was conducted.
• An institutional PP-LLM (Claude Sonnet 3.5) augmented forms by mining EMRs and suggesting protocols.
• Clinical information and protocoling accuracy were compared against a consensus reference standard established by expert radiologists.

Main Results:

• PP-LLM-augmented forms showed superior clinical information, with only 0.4%-0.6% rated deficient compared to 12.4%-22.6% for original forms (p < 0.001).
• Inter-rater reliability for LLM-enhanced requests was almost perfect (AC1 = 0.99) versus substantial for original forms (AC1 = 0.62).
• PP-LLM protocoling accuracy was 95.8%, significantly higher than human radiologists (88.6% and 88.2%, p < 0.001).

Conclusions:

• LLM augmentation of musculoskeletal MRI request forms yields superior clinical information and protocoling accuracy.
• Institutional adoption of PP-LLMs can improve MRI appropriateness and enhance patient care.