An open-source fine-tuned large language model for radiological impression generation: a multi-reader performance...

Area of Science:

• Artificial Intelligence in Medicine
• Radiology Informatics

Background:

• Radiology report impressions are crucial but can be subjective.
• There is a need for automated methods to generate consistent impressions.

Purpose of the Study:

• To fine-tune and evaluate an open-source Large Language Model (LLM) for automatic generation of radiology report impressions.
• To assess the LLM's performance across different imaging modalities and healthcare institutions.

Main Methods:

• A retrospective study utilized a large dataset of CT, US, and MRI reports from two hospitals.
• The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score was used for automatic evaluation.
• A reader study with five subspecialist radiologists evaluated clinical accuracy, grammar, and style.

Main Results:

• The LLM achieved notable ROUGE-L scores across modalities, indicating substantial overlap with human-written impressions.
• Reader study showed LLM impressions scored highly in clinical accuracy (3.56/4), grammatical accuracy (3.92/4), and stylistic quality (3.37/4).
• LLM performance was highest for acute findings and shorter impressions, with a slight degradation on external validation.

Conclusions:

• A fine-tuned open-source LLM can generate radiology report impressions with acceptable clinical accuracy, grammar, and style.
• LLMs show potential for drafting impressions, aiding in streamlining radiologist workflows.