Large language models for error detection in radiology reports: a comparative analysis between closed-source and...

Area of Science:

• Artificial Intelligence in Medical Imaging
• Natural Language Processing in Healthcare
• Radiology Report Analysis

Background:

• Large language models (LLMs) can aid in radiology report error detection.
• Privacy concerns limit the clinical use of closed-source LLMs.
• This study evaluates open-source LLMs as a privacy-compliant alternative.

Purpose of the Study:

• To compare the error detection performance of open-source LLMs against closed-source LLMs in radiology reports.
• To assess the clinical applicability of privacy-compliant LLMs for ensuring accuracy in diagnostic reporting.

Main Methods:

• 120 radiology reports (X-ray, ultrasound, CT, MRI) were analyzed.
• 397 errors were inserted into 100 reports across five categories.
• Two open-source (Llama 3-70b, Mixtral 8x22b) and two closed-source (GPT-4, GPT-4o) LLMs were evaluated.

Main Results:

• Open-source LLMs processed reports faster (6s ± 2s) than closed-source LLMs (13s ± 4s).
• Closed-source LLMs achieved higher detection rates (GPT-4o: 88%, GPT-4: 83%) than open-source models (Llama 3-70b: 79%, Mixtral 8x22b: 73%).
• Numerical errors were detected more frequently (88%) than typographical (75%) or interpretation errors (70%).

Conclusions:

• Open-source LLMs offer effective, privacy-compliant error detection in radiology reports.
• While slightly less accurate than closed-source models, open-source LLMs have significant clinical potential.
• Further development of open-source LLMs can enhance accuracy and improve patient care through reliable reporting.