Multimodal Large Language Models for Breast Ultrasound Report Auditing: Workflow-Error Detection, False-Positive...

Area of Science:

• Radiology and Medical Imaging
• Artificial Intelligence in Healthcare
• Medical Informatics

Background:

• Large language models (LLMs) show promise for radiology reporting.
• The utility of integrating key images into LLM reports for breast ultrasound auditing is not well-established.

Purpose of the Study:

• To evaluate the incremental value of multimodal input (text + key images) for LLM-based breast ultrasound report auditing.
• To compare LLM performance with and without key images against a physician benchmark.

Main Methods:

• Retrospective analysis of 818 breast ultrasound exams.
• Construction of a 300-report subset enriched with 329 errors.
• Evaluation of GPT-5.5 and Gemini 3.1 Pro Preview using text-only and multimodal inputs.
• Comparison with a physician reader's performance.

Main Results:

• Multimodal input significantly improved LLM sensitivity and error recall, particularly for image-related errors.
• Gemini exhibited false positives, mainly due to marker misinterpretation.
• Physician performance exceeded LLMs in sensitivity and error recall, with no false positives.
• Key-image input underperformed physician-interpreted findings for malignancy and BI-RADS classification.

Conclusions:

• Multimodal input enhances LLM-based detection of breast ultrasound workflow errors, especially those involving visual cues.
• LLM performance, even with multimodal input, requires careful consideration due to overcalling and lower accuracy compared to expert physicians.
• Further research is needed to optimize LLM integration for reliable clinical application in breast ultrasound reporting.