Artificial intelligence vs. emergency physicians: who diagnoses better?

Area of Science:

• Emergency Medicine
• Artificial Intelligence
• Medical Diagnostics

Background:

• Emergency departments face diagnostic challenges.
• Large Language Models (LLMs) are emerging AI tools with potential clinical applications.
• Evaluating LLM performance against human experts is crucial for adoption.

Purpose of the Study:

• To compare diagnostic accuracy of LLMs versus an emergency medicine specialist.
• To assess LLM capabilities in selecting initial diagnostic tests.
• To evaluate LLMs in simulated emergency department scenarios.

Main Methods:

• Created simulated emergency case presentations (history, exam findings).
• Compared diagnostic accuracy and test selection of a human expert and three LLMs (ChatGPT-4, 4o, 3.5-mini).
• Assessed accuracy based on predefined correct diagnoses and appropriate first-line tests.

Main Results:

• LLMs achieved higher diagnostic accuracy (97-99%) than the human expert (92%).
• LLMs performed comparably to the human expert in initial diagnostic test selection (80-89% vs. 88%).
• Most diagnostic errors involved cardiovascular and gastrointestinal cases.

Conclusions:

• LLMs demonstrate acceptable diagnostic accuracy, exceeding human expert performance.
• LLMs show comparable performance to human experts in initial test selection.
• AI models show promise as decision-support tools in emergency medicine, warranting further real-world validation.