Evaluating ChatGPT's adherence to evidence-based heart failure guidelines: a comparative analysis using the 2023 ESC...

Heart failure (HF) remains a major cause of morbidity and mortality worldwide. Large language models (LLMs) such as ChatGPT are emerging as potential clinical decision support tools, but their adherence to specialty guidelines is not well characterised.

To evaluate the accuracy and guideline concordance of ChatGPT-5 in managing real-world HF scenarios compared with the 2023 European Society of Cardiology (ESC) and 2022 American College of Cardiology (ACC)/American Heart Association (AHA)/Heart Failure Society of America (HFSA) recommendations.

Thirty-eight anonymised HF clinical vignettes spanning reduced, mildly reduced, and preserved ejection fraction phenotypes and varied New York Heart Association (NYHA) classes were presented to ChatGPT-5. Two board-certified cardiologists independently graded each response for concordance with guideline recommendations using a 4-point scale (3 = fully concordant, 2 = partially concordant, 1 = discordant, 0 = unsafe/harmful). Discrepancies were adjudicated by a third reviewer. Descriptive statistics summarised performance and inter-rater agreement.

Of the 38 responses, 20 (53%) were fully concordant, 4 (11%) partially concordant, 8 (21%) discordant, and 6 (16%) unsafe/harmful. Most inaccuracies involved vague drug titration guidance, incomplete device therapy recommendations, or omission of guideline-directed medical therapy (GDMT). Unsafe suggestions occurred in complex device or advanced therapy decisions. Inter-rater agreement was high.

ChatGPT-5 showed moderate concordance with ESC and ACC/AHA/HFSA HF guidelines, indicating potential value as a tool for knowledge synthesis and preliminary clinical support. However, its outputs require expert validation, and safe clinical integration will depend on future models incorporating guideline-based frameworks, real-time data, and rigorous physician oversight.