Evaluating GPT-4's visual interpretation and clinical reasoning on emergency settings: A 5-year analysis

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Emergency Medicine Diagnostics

Background:

• Generative AI, specifically large language models (LLMs) like GPT-4, is increasingly utilized in medical education.
• This study assesses GPT-4's performance in interpreting emergency medicine board exam questions, including both text and image-based formats.

Purpose of the Study:

• To evaluate the cognitive and decision-making capabilities of GPT-4 in simulated emergency medicine scenarios.
• To determine GPT-4's accuracy and reasoning abilities on board exam questions.

Main Methods:

• An observational study utilized Taiwan Emergency Medicine Board Exam questions from 2018-2022.
• GPT-4's performance was analyzed for accuracy and reasoning across different question types.
• Statistical analyses explored factors influencing performance, such as knowledge dimension and cognitive level.

Main Results:

• GPT-4 achieved 60.1% overall accuracy, with similar performance on text-based (60.2%) and image-based (59.3%) questions.
• High accuracy was observed in image identification (100%) and interpretation (86.4%), but declined in diagnostic reasoning (83.1%) and decision-making (59.3%).
• No significant associations were found between question characteristics and GPT-4's performance.

Conclusions:

• GPT-4 exhibits strong image recognition and moderate diagnostic reasoning but limited clinical decision-making, particularly when integrating visual and clinical data.
• While promising as an educational tool, GPT-4's current reliance on pattern recognition over clinical understanding limits its real-world applicability in emergency medicine.
• Further advancements are necessary for AI to reliably support emergency medical decision-making.