Comparison of a Specialized Large Language Model with GPT-4o for CT and MRI Radiology Report Summarization

Area of Science:

• Radiology
• Artificial Intelligence
• Natural Language Processing

Background:

• General-purpose large language models (LLMs) like GPT-4o show potential in radiology.
• The comparative performance of specialized LLMs versus general-purpose LLMs for radiology report summarization is not well-established.

Purpose of the Study:

• To compare the summarization performance of a specialized LLM (LLM-RadSum) against GPT-4o for radiology reports.
• To evaluate the factual consistency, coherence, safety, and clinical utility of generated summaries.

Main Methods:

• Developed LLM-RadSum using a retrospective hospital dataset (training/internal test sets).
• Evaluated F1 scores on internal and external test sets (CT/MRI reports).
• Conducted human evaluation on 1800 reports, comparing LLM-RadSum and GPT-4o outputs using criteria like factual consistency and clinical safety.

Main Results:

• LLM-RadSum achieved higher median F1 scores (0.58) on human evaluation compared to GPT-4o (0.30; P < .001).
• Over 81.5% of LLM-RadSum outputs met expert standards, while 27.8% of GPT-4o outputs required adjustments.
• LLM-RadSum showed superior performance across various modalities, anatomic regions, and patient demographics.

Conclusions:

• A specialized LLM (LLM-RadSum) significantly outperforms general-purpose GPT-4o in summarizing radiology reports.
• LLM-RadSum provides more factually consistent, coherent, safe, and clinically useful summaries.
• Specialized LLMs are more effective for specific medical NLP tasks like radiology report summarization.