Large-scale evaluation of multimodal large language models for pneumothorax detection

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Imaging Analysis

Background:

• Pneumothorax detection on chest X-rays (CXRs) is critical in acute care.
• Advanced AI models offer potential for image interpretation, but their diagnostic reliability needs evaluation.
• This study assesses state-of-the-art multimodal models for pneumothorax detection.

Purpose of the Study:

• To evaluate the diagnostic performance of three leading multimodal AI systems (GPT-4o, Gemini 2 Pro, Claude 4 Sonnet) in detecting pneumothorax.
• To analyze performance across different pneumothorax sizes using a large, annotated CXR dataset.

Main Methods:

• Analysis of 10,675 CXRs from the SIIM-ACR Pneumothorax Segmentation dataset.
• Evaluation of three multimodal models using a uniform, image-based approach.
• Comparison of model outputs against reference results for accuracy, sensitivity, specificity, precision, and F1 scores, with subgroup analyses by pneumothorax size.

Main Results:

• Pneumothorax prevalence was 22.3%.
• All models achieved high specificity (>0.90) but low sensitivity (0.16-0.36).
• Gemini 2 showed the best overall accuracy (0.79) and specificity (0.95); Claude 4 demonstrated higher sensitivity. Performance improved with larger pneumothorax sizes, but small lesions remained challenging. Statistically significant differences were found between models (P < 0.050).

Conclusions:

• Current multimodal large language models (LLMs) demonstrate strong reliability in identifying normal CXRs but have limited ability to detect small or subtle pneumothoraxes.
• Low sensitivity restricts their use as standalone tools for excluding pneumothorax.
• Future refinements may enable these AI models to support radiologists by enhancing workflow efficiency and diagnostic confidence.