Limitations in Chest X-Ray Interpretation by Vision-Capable Large Language Models, Gemini 1.0, Gemini 1.5 Pro, GPT-4...

Area of Science:

• Artificial Intelligence in Medical Imaging
• Computer Vision for Diagnostic Radiology

Background:

• Accurate chest X-ray (CXR) interpretation is crucial for diagnosis, requiring identification of lesions, diagnosis, location, size, and number.
• The efficacy of vision-capable large language models (LLMs) for image-only CXR interpretation, without clinical context, is not well-established.

Purpose of the Study:

• To evaluate the performance of leading LLMs in interpreting CXRs based solely on image data.
• To assess the accuracy of LLMs in primary diagnosis and identification of key imaging features across various pulmonary conditions.

Main Methods:

• 247 CXRs across 13 diagnostic categories were analyzed by Gemini 1.0, Gemini 1.5 Pro, GPT-4 Turbo, and GPT-4o.
• LLM outputs were evaluated for primary diagnosis accuracy (fully correct, partially correct, incorrect) and key feature identification.
• Non-diagnostic features like imaging views and devices were analyzed separately.

Main Results:

• LLMs demonstrated higher sensitivity for detecting large, bilateral, and multiple lesions, as well as prominent devices (e.g., pulmonary edema, pacemakers).
• Gemini 1.5 Pro achieved the highest overall detection rate, followed by Gemini 1.0, GPT-4o, and GPT-4 Turbo.
• Model performance varied in describing imaging features like views and markers, with some limitations in recognizing specific medical terminology.

Conclusions:

• LLMs can identify certain CXR diagnoses and features, particularly for significant abnormalities.
• Limitations persist in detecting small lesions, determining laterality, and complex diagnostic reasoning.
• Further advancements are necessary for LLMs to reliably interpret CXRs without accompanying clinical information.