Radiologically Relevant Clinical History Summarization with Large Language Models: A Multireader Performance Study

Area of Science:

• Artificial Intelligence in Radiology
• Clinical Decision Support Systems
• Natural Language Processing in Healthcare

Background:

• Incomplete clinical histories accompanying imaging orders can compromise diagnostic accuracy and workflow efficiency.
• Accurate and comprehensive imaging indications are crucial for appropriate protocol selection and diagnostic focus.

Purpose of the Study:

• To evaluate the effectiveness of large language models (LLMs) in improving the clinical utility of imaging indications.
• To assess if LLMs can leverage clinical notes to generate more comprehensive and factual indications.

Main Methods:

• A retrospective study curated a dataset of radiology reports with paired clinician and radiologist indications linked to clinical notes.
• Twenty radiologists compared indications from referring clinicians and best-performing LLMs on comprehensiveness, factuality, and usefulness for protocoling and interpretation.
• Statistical analysis using cumulative link mixed models compared LLM-generated indications against clinician-provided ones.

Main Results:

• LLM-generated indications were rated as more comprehensive and factual than referring clinician indications.
• A proprietary LLM's indications were ranked highest in usefulness for protocoling, interpretation, and overall utility.
• Comprehensiveness of indications was the strongest factor influencing overall rankings.

Conclusions:

• Large language models can generate radiology-relevant indications from clinical notes that surpass clinician-provided indications in quality.
• AI-powered indications, particularly from proprietary LLMs, offer significant potential to enhance radiology protocoling and interpretation.
• The study highlights the value of LLMs in improving diagnostic accuracy and workflow efficiency in medical imaging.