Benchmarking vision-language models for diagnostics in emergency and critical care settings

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Clinical Decision Support Systems

Background:

• Vision-language models (VLMs) are emerging AI tools with potential in healthcare.
• Their application in high-acuity clinical settings like emergency and intensive care units is not well understood.
• Current open-source VLMs may not meet the diagnostic demands of acute care.

Purpose of the Study:

• To evaluate the diagnostic performance of small open-source VLMs in acute care scenarios.
• To compare the capabilities of open-source VLMs against a state-of-the-art model (GPT-4o).
• To identify limitations and areas for improvement in VLMs for emergency medicine.

Main Methods:

• Utilized a multimodal dataset comprising diagnostic questions with medical images and clinical context.
• Benchmarked several small, open-source vision-language models.
• Compared performance against the advanced GPT-4o model.

Main Results:

• Open-source VLMs achieved a maximum diagnostic accuracy of 40.4%.
• GPT-4o significantly outperformed open-source models, reaching 68.1% accuracy.
• A substantial performance gap exists between current open-source VLMs and advanced models in acute care.

Conclusions:

• Open-source VLMs currently exhibit insufficient diagnostic accuracy for critical acute care applications.
• Significant advancements in specialized training and model optimization are required for open-source VLMs.
• GPT-4o shows promise but further validation is needed for clinical deployment in emergency and intensive care.