Fine-Tuned Large Language Model for Automated Radiology Impression Generation: A Multicenter Evaluation

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Natural Language Processing for Healthcare

Background:

• Radiology reporting is crucial for patient care.
• Current reporting methods can be time-consuming and prone to variability.
• Large language models (LLMs) offer potential for automating and improving report generation.

Purpose of the Study:

• To develop and evaluate a fine-tuned LLM, the Medical Imaging Report Assistant (MIRA).
• To assess MIRA's performance in generating radiology impressions from multicenter data.
• To measure MIRA's accuracy, reporting efficiency, and clinical applicability compared to existing models.

Main Methods:

• A retrospective multicenter dataset of 1.87 million radiology reports was compiled.
• A large language model was fine-tuned using a prompt-based strategy.
• Performance was evaluated using automated metrics and blinded comparisons by 24 radiologists.

Main Results:

• Site/modality-aware prompting significantly improved similarity scores (BERTScore-F/Sentence Similarity: 0.92/0.92 internally, 0.82/0.80 externally).
• Human evaluation showed MIRA outperformed GPT-4o in similarity and F1 score (P < .001).
• MIRA-generated impressions were rated as good as reference impressions in 69.0% of cases, reducing draft time by 0.46 min and increasing interradiologist agreement (P < .001).

Conclusions:

• MIRA, a fine-tuned LLM, effectively generates clinically aligned radiology impressions.
• The model demonstrates improved accuracy, efficiency, and reporting consistency in multicenter settings.
• MIRA shows significant potential to enhance the radiology reporting workflow.