Opportunities and Challenges of Visual Large Language Models in Imaging Diagnostics: Lessons from Brain Metastasis...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Radiology

Background:

• Evaluating diagnostic accuracy of visual large language models (vLLMs) for brain metastases detection.
• Assessing GPT-4o and Claude Sonnet 3.5 using combined imaging and textual MRI data.
• Investigating vLLM performance on routine MRI scans for oncological patients.

Purpose of the Study:

• To determine the diagnostic accuracy of GPT-4o and Claude Sonnet 3.5 in identifying brain metastases.
• To compare the performance of these vLLMs in terms of sensitivity, specificity, and other diagnostic metrics.
• To evaluate the reliability of vLLMs in generating radiological reports for brain metastases.

Main Methods:

• Retrospective analysis of 100 MRI examinations (50 with, 50 without brain metastases) from 77 patients.
• Inputting single representative slices per sequence with clinical history to GPT-4o and Claude Sonnet 3.5.
• Evaluating vLLM-generated reports for detection accuracy, overdiagnosis, sequence recognition, localization, laterality, and size estimation.

Main Results:

• Both vLLMs achieved 100% sensitivity but very low specificity (GPT-4o: 8%, Sonnet 3.5: 4%).
• Diagnostic accuracy was low (GPT-4o: 54%, Sonnet 3.5: 52%), with both models hallucinating lesions in 12% of cases.
• Sequence identification was high, but anatomical localization and lesion laterality showed limitations.

Conclusions:

• GPT-4o and Claude Sonnet 3.5 demonstrate high sensitivity for brain metastases detection but are currently unsuitable for clinical use due to low specificity and hallucinations.
• The spatial reliability and diagnostic behavior of vLLMs require further investigation, particularly concerning the balance of visual and textual input.
• Future research should focus on improving the specificity and reliability of vLLMs for accurate radiological reporting.