Augmenting clinical decision-making with large language models: evaluation across general and specialty tasks

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Informatics

Background:

• Previous studies on large language models (LLMs) in clinical settings primarily focused on their standalone capabilities.
• There is a need to evaluate how LLMs can actively support clinicians in real-world practice across diverse medical contexts.

Purpose of the Study:

• To assess the impact of LLM assistance on clinician performance across different medical specialties, disease types, experience levels, and stages of the clinical decision-making process.
• To compare the effectiveness of different LLMs in supporting clinical tasks.

Main Methods:

• Three LLMs (Deepseek-R1, GPT-4o-mini, LLaMA-4) were evaluated using general-disease tasks and a specific prostate cancer clinical workflow.
• Clinicians of varying seniority performed tasks independently and then with LLM assistance, with performance rated by experts.
• Statistical analysis, including the Wilcoxon signed-rank test, was used to compare performance metrics.

Main Results:

• LLM assistance significantly improved clinician performance in general disease tasks (P < .05).
• Junior clinicians showed substantial performance gains (15.9%-20.8%) across all clinical stages with LLM support, surpassing senior clinicians' unaided performance in later stages.
• Performance improvements varied by LLM, with Deepseek-R1 excelling in diagnostic tasks.

Conclusions:

• LLM assistance offers greater benefits to less-experienced clinicians, particularly in downstream decision-making stages.
• LLMs show promise as clinical decision support tools, capable of improving decision quality and mitigating performance disparities related to clinician experience.