Generative Large Language Models Trained for Detecting Errors in Radiology Reports

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Radiology Reporting

Background:

• Large language models (LLMs) show potential for medical proofreading, but their use in radiology report error detection is limited.
• Developing AI tools for accurate radiology report analysis is crucial for patient safety and clinical decision-making.

Purpose of the Study:

• To develop and evaluate generative LLMs for detecting various errors in radiology reports.
• To assess the performance of different LLMs and prompting strategies in medical proofreading tasks.

Main Methods:

• A dataset of synthetic and real-world radiology reports (MIMIC-CXR) was created, including error-free and erroneous reports.
• Errors were categorized into negation, left/right, interval change, and transcription types.
• Models including Llama-3, GPT-4, and BiomedBERT were fine-tuned using zero-shot, few-shot, and fine-tuning strategies, with performance evaluated by F1 scores and radiologist review.

Main Results:

• The fine-tuned Llama-3-70B-Instruct model achieved the highest overall F1 score of 0.780, with specific high scores for transcription errors (0.828).
• Radiologist review confirmed the model's ability to detect errors, with 99 reports confirmed by both reviewers and 163 by at least one reviewer.

Conclusions:

• Generative LLMs, when fine-tuned on diverse radiology report datasets, substantially enhance the accuracy of medical proofreading.
• These AI models offer a promising solution for improving the quality and reliability of radiology reports.