Augmenting clinical decision-making with large language models: evaluation across general and specialty tasks

Large language models (LLMs) have shown promise in clinical applications, yet prior studies mainly evaluated their standalone performance on benchmarks or examinations. This study assesses how LLMs support clinicians across specialties, disease contexts, experience levels and decision-making stages.

We evaluated 3 LLMs (Deepseek-R1, GPT-4o-mini and LLaMA-4) using 2 task types: (1) general-disease tasks spanning specialties and disease incidence levels, and (2) prostate disease scenario covering the clinical workflow, including diagnosis, treatment planning, postoperative rehabilitation and prognosis. Clinicians with different seniority completed tasks independently and repeated them after reviewing LLM-generated responses. Performance was rated by experts using a 5-point Likert scale, and differences were analyzed with the Wilcoxon signed-rank test.

LLM assistance significantly improved clinician performance across general disease tasks (P < .05). In the specialty scenario, junior clinicians' scores increased by 0.579-0.723 (15.9%-20.8%) across 4 clinical stages, whereas senior clinicians' improvements ranged from 0.053-0.303 (1.3%-7.5%). With LLM support, junior clinicians surpassed senior clinicians' unaided performance in postoperative rehabilitation and prognosis stages (P < .05) and achieved comparable performance in diagnosis and treatment stages (P > .05). Performance varied among models, with Deepseek-R1 performing best in diagnostic tasks.

These findings suggest that LLM assistance may provide greater benefit to less-experienced clinicians and are particularly effective in downstream decision-making stages, indicating a potential role in mitigating experience-related performance disparities.

LLMs may serve as supportive clinical-decision support tools across diseases, specialties, clinician experience levels, and workflow stages, with observed improvements in decision quality and reduced performance disparities.