Evaluating Large Language Models in Cardiovascular Antithrombotic Care: Performance, Accuracy, and Implications for...

Large language models (LLMs) are increasingly accessible for medical decision making and are often used by medical practitioners and patients. However, previous studies have raised concerns about the accuracy of individual LLMs in assisting with patient management.

This study assessed the performance of 7 publicly available LLMs on validated cardiovascular antithrombotic care scenarios, evaluated by 3 independent clinicians for accuracy and reasoning. The results were compared with the performance of volunteer clinicians, based on a survey conducted at the Canadian Cardiovascular Congress in 2023. Statistical analyses ensured interobserver reliability and evaluated performance differences among models.

Claude 3 Opus correctly answered 85% of clinical scenarios, significantly outperforming both other LLMs (P = < 0.001) and all clinician groups. Among clinicians, cardiologists, and senior residents achieved the highest accuracy rates: 43% (95 confidence interval [CI], 32%-52%) and 47% (95 CI, 39%-56%) respectively, comparable with GPT-4o (55%) and Claude 3.5 Sonnet (44%). General practitioners performed similarly to Claude 3 Sonnet and Gemini 1.5: 22% (95 CI, 11%-33%) vs 26% vs 30%, whereas medical students achieved 8.3% (95 CI, 2%-15%), closely aligning with GPT-3.5 (10%).

The performance of LLMs in cardiovascular clinical scenarios varied widely, with some models outperforming clinicians, and some free-tier models providing inappropriate medical advice to clinicians. However, all tested models demonstrated acceptable performance for delivering patient advice regarding lifestyle and dietary recommendations. Clinicians and patients should exercise caution when using LLMs, select the best LLM for the task, and crosscheck provided references to ensure safe use of LLMs in practice.

NCT05923658.