Out-of-the-Box Large Language Models for Detecting and Classifying Critical Findings in Radiology Reports Using...

Area of Science:

• Artificial Intelligence in Radiology
• Natural Language Processing in Healthcare

Background:

• Radiology reports are increasing in complexity and volume, challenging timely critical finding communication.
• Effective methods are needed to extract critical information from radiology reports efficiently.

Purpose of the Study:

• To evaluate the performance of two large language models (LLMs), GPT-4 and Mistral-7B, in detecting and classifying critical findings in radiology reports.
• To compare the effectiveness of various prompt strategies for LLM-based analysis of radiology reports.

Main Methods:

• 252 radiology reports from MIMIC-III and 180 chest radiography reports from CheXpert Plus were analyzed.
• Prompt engineering was performed, and a final prompt was selected for zero-shot, few-shot static, and few-shot dynamic prompting.
• GPT-4 and Mistral-7B processed test sets, with evaluation using automated metrics (BLEU-1, ROUGE-F1, G-Eval) and manual metrics (precision, recall).

Main Results:

• Few-shot static prompting with GPT-4 achieved the highest ROUGE-F1 score (0.797) for true critical findings.
• GPT-4 demonstrated superior precision and recall compared to Mistral-7B across both holdout and external test sets for all critical finding categories.
• GPT-4 achieved 90.1% precision and 86.9% recall for true critical findings in the holdout test set.

Conclusions:

• Out-of-the-box LLMs can effectively detect and classify critical findings in radiology reports.
• A few-shot static prompting strategy is optimal for detecting true critical findings using LLMs.
• General-purpose LLMs show promise in adapting to specialized medical tasks with minimal data annotation.