Diagnostic Performance of Multimodal Large Language Models for Central Venous Catheter Assessment Chest Radiographs...

Area of Science:

• Medical imaging analysis
• Artificial intelligence in healthcare
• Critical care medicine

Background:

• Chest radiography is standard for assessing central venous catheter (CVC) placement in intensive care units.
• The reliability of multimodal large language models (MLLMs) for interpreting these radiographs is not well-established.
• This study evaluates MLLMs against intensivists for CVC assessment and pneumothorax detection.

Purpose of the Study:

• To compare the diagnostic performance of MLLMs and human intensivists in evaluating CVC placement on chest radiographs.
• To assess the accuracy of MLLMs and intensivists in identifying CVC tip position and detecting pneumothorax.
• To determine the reliability of MLLMs for critical radiography tasks.

Main Methods:

• A retrospective study analyzed 183 chest radiographs of critically ill adult patients post-CVC placement.
• Four intensivists and five MLLMs independently evaluated the radiographs.
• Radiologist consensus served as the reference standard, with performance metrics including accuracy, sensitivity, specificity, and balanced accuracy.

Main Results:

• Intensivists performed well in CVC access classification but showed variable performance in tip assessment.
• MLLMs exhibited a wide range of accuracy for CVC access (0.339-0.874) and poor specificity for tip assessment.
• MLLMs failed to detect pneumothorax, classifying all cases as negative; intensivists outperformed MLLMs in CVC access and tip assessment.

Conclusions:

• The evaluated MLLMs showed unreliable diagnostic performance compared to experienced intensivists.
• MLLM performance was skewed by class imbalance and a tendency towards dominant responses.
• Task-specific validation and comprehensive reporting are essential for MLLMs in medical imaging.