Large Language Model Performance and Clinical Reasoning Tasks

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing in Healthcare

Background:

• Large language models (LLMs) are increasingly marketed for clinical applications.
• Current evaluations using multiple-choice tests do not fully capture clinical reasoning complexity.
• The ability of LLMs to replicate full-spectrum clinical reasoning remains uncertain.

Purpose of the Study:

• To evaluate the longitudinal clinical reasoning ability of state-of-the-art LLMs.
• To introduce a multidimensional, clinically meaningful benchmark for clinical-grade artificial intelligence (AI).

Main Methods:

• A cross-sectional study evaluated 21 off-the-shelf LLMs using standardized clinical vignettes.
• Performance was assessed across five domains: differential diagnosis, diagnostic testing, final diagnosis, management, and miscellaneous reasoning.
• The Proportional Index of Medical Evaluation for LLMs (PrIME-LLM) score was the primary outcome measure.

Main Results:

• PrIME-LLM scores varied, with Grok 4 performing highest (0.78) and Gemini 1.5 Flash lowest (0.64).
• Reasoning-optimized models outperformed non-reasoning models; GPT models scored highest overall.
• Models struggled with differential diagnosis (failure rates >0.80) but excelled in final diagnosis (failure rates <0.40). Multimodal performance improved accuracy with image inputs.

Conclusions:

• Frontier LLMs demonstrate high accuracy in final diagnoses but exhibit significant weaknesses in differential diagnosis and managing uncertainty.
• The PrIME-LLM framework reveals critical reasoning gaps not apparent with traditional benchmarks.
• Despite improvements, current LLMs lack the advanced clinical reasoning necessary for safe deployment.