Performance of Multimodal Large Language Models in Detection and Position Assessment of Thoracic Devices on Chest...

Area of Science:

• Medical Imaging AI
• Artificial Intelligence in Healthcare
• Radiology AI

Background:

• Accurate identification and positioning of thoracic devices on chest radiographs are crucial for patient safety in intensive care settings.
• Multimodal large language models (LLMs) present a potential for automated evaluation, but their efficacy in this specific domain remains largely unexamined.

Purpose of the Study:

• To evaluate the performance of three leading multimodal LLMs in identifying the presence and positioning of critical thoracic devices on chest radiographs.
• To compare LLM performance against board-certified radiologists and assess the reliability of these AI models for clinical application.

Main Methods:

• Three multimodal LLMs (GPT-4o, Gemini 3.1 Flash Lite Preview, Claude Sonnet 4.6) were tested on 4813 chest radiographs from the RANZCR CLiP dataset for device presence and positioning.
• Performance metrics included balanced accuracy, MCC, and kappa statistics, with additional analyses involving a reader study with radiologists, external validation, prompt sensitivity, and error taxonomy.

Main Results:

• LLMs demonstrated variable performance for device presence and uniformly poor sensitivity for abnormal positioning (MCC ≤ 0.028).
• Board-certified radiologists significantly outperformed all evaluated LLMs in paired comparisons.
• LLM performance showed instability due to prompt variations and inference runs, with systematic failures in multi-device cases.

Conclusions:

• General-purpose multimodal LLMs are currently unreliable for autonomous thoracic device assessment in clinical practice.
• These AI models may serve a limited role as adjunct screening tools for device presence, but not for positioning assessment.
• Findings do not extend to specialized, regulatory-approved clinical AI systems designed for medical imaging tasks.