Evaluating Large Language Models in Cardiovascular Antithrombotic Care: Performance, Accuracy, and Implications for...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Cardiovascular Medicine

Background:

• Large language models (LLMs) are increasingly used in healthcare, but concerns exist regarding their accuracy in patient management.
• Previous studies have highlighted potential inaccuracies of individual LLMs in clinical settings.

Purpose of the Study:

• To assess the performance of publicly available large language models (LLMs) in cardiovascular antithrombotic care scenarios.
• To compare LLM performance against human clinicians in simulated clinical decision-making.

Main Methods:

• Seven publicly available LLMs were evaluated on validated cardiovascular antithrombotic care scenarios.
• Three independent clinicians assessed LLM accuracy and reasoning, comparing results to a survey of volunteer clinicians.
• Statistical analyses were used to ensure interobserver reliability and evaluate performance differences.

Main Results:

• Claude 3 Opus achieved 85% accuracy, significantly outperforming other LLMs and all clinician groups.
• Cardiologists and senior residents showed the highest clinician accuracy (43-47%), comparable to GPT-4o (55%) and Claude 3.5 Sonnet (44%).
• General practitioners and medical students performed less accurately, with some free-tier models providing inappropriate advice.

Conclusions:

• LLM performance in cardiovascular scenarios varies widely, with some models surpassing clinician accuracy.
• All tested LLMs demonstrated acceptable performance for lifestyle and dietary recommendations.
• Clinicians and patients should use LLMs cautiously, select appropriate models, and verify information for safe practice.