Can AI match emergency physicians in managing common emergency cases? A comparative performance evaluation

Area of Science:

• Emergency Medicine
• Artificial Intelligence
• Clinical Decision Support

Background:

• Large language models (LLMs) are being explored for clinical decision support.
• Their utility in high-stakes emergency medicine scenarios is not well-understood.

Purpose of the Study:

• To compare the diagnostic and therapeutic accuracy of ChatGPT (GPT-4) against a board-certified emergency physician.
• To evaluate performance across diverse emergency medicine cases.

Main Methods:

• A comparative study using 15 standardized emergency scenarios.
• Independent evaluation of cases by ChatGPT and a physician on diagnosis, investigations, treatment, safety, and complexity.
• Scoring out of 5, with concordance categorized as high, moderate, or low.

Main Results:

• ChatGPT achieved high concordance (5/5) in 53.3% of cases.
• Performance was strongest in structured conditions (e.g., STEMI, DKA) and weaker in complex cases (e.g., stroke, trauma).
• Moderate (26.7%) and low (20.0%) concordance were observed in other scenarios.

Conclusions:

• ChatGPT demonstrated strong alignment with physician decisions in structured emergency scenarios.
• Reliability was limited in complex and less defined emergency cases.
• AI tools can serve as valuable educational and supportive resources but do not replace human clinical judgment.