Evaluating large language and large reasoning models as decision support tools in emergency internal medicine

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Informatics

Background:

• Large Language Models (LLMs) show potential for clinical decision support but exhibit variable real-world performance.
• Comparison of three leading LLMs (OpenAI's o1, Anthropic's Claude-3.5-Sonnet, Meta's Llama-3.2-70B) against human experts was conducted.
• The study focused on an emergency internal medicine setting.

Purpose of the Study:

• To compare the clinical performance of leading LLMs against human experts.
• To evaluate LLM-generated reports for diagnostic accuracy, therapy planning, and follow-up recommendations.
• To assess the utility of advanced LLMs as clinical decision-support tools in real-world emergency cases.

Main Methods:

• A prospective comparative study involving 73 anonymized patient cases from an Emergency Internal Medicine ward.
• Two independent internal medicine specialists evaluated LLM and human-authored reports.
• Evaluation included diagnostic test relevance, final diagnosis, therapy plan, and follow-up, using Likert scales and statistical tests.

Main Results:

• The o1 LLM achieved performance statistically indistinguishable from human physicians (mean rating 3.63 vs. 3.67).
• Claude-3.5-Sonnet (3.38) and Llama-3.2-70B (3.23) scored significantly lower, with noted errors in therapy planning.
• All LLMs demonstrated ≥90% accuracy in final diagnoses and admission decisions, with o1 achieving 100% accuracy in classifying abnormal lab values.

Conclusions:

• An advanced LLM (o1) with enhanced reasoning capabilities can match expert-level clinical performance in emergency medicine.
• LLMs show significant potential as valuable clinical decision-support tools.
• Further research into LLM applications in healthcare is warranted.