Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation

Large language models (LLMs) are increasingly applied in interventional cardiology, but hallucinations limit their clinical utility.

The aim of this study was to assess whether retrieval-augmented generation (RAG), a technique that allows LLMs to access guideline content during response generation, improves accuracy when answering questions on the basis of the guidelines for acute coronary syndromes.

The accuracy of ChatGPT-4o, DeepSeek R1, and Med-PaLM 2 was compared using a set of 38 open-ended cardiology guideline-based questions and answers. ChatGPT-4o and DeepSeek R1 were evaluated with and without RAG, while Med-PaLM 2, a medicine-specific LLM, was tested without RAG. Model outputs were compared against guideline recommendations using an artificial intelligence-powered similarity score tool.

DeepSeek R1 with RAG achieved the highest accuracy (94.7%; 95% CI: 82.7%-98.5%), followed by ChatGPT-4o with RAG (92.1%; 95% CI: 79.2%-97.3%) (P = 0.922). ChatGPT-4o without RAG achieved 71.1% accuracy (95% CI: 55.2%-83.0%), which significantly improved with RAG (P = 0.017). Among non-RAG models, DeepSeek R1 demonstrated the highest accuracy (78.9%; 95% CI: 63.7%-88.9%), followed by ChatGPT-4o without RAG (71.1%) (P = 0.083). Med-PaLM 2 had the lowest accuracy (68.4%; 95% CI: 52.5%-80.9%). Spearman correlation analysis revealed a strong correlation between DeepSeek R1 without RAG and Med-PaLM 2 (r = 0.646; 95% CI: 0.411-0.800; P < 0.001), indicating similar response patterns. Scatterplot analysis further revealed that RAG disproportionately improved lower scoring questions in DeepSeek R1 while improving scores more evenly in ChatGPT-4o.

Embedding guideline content into LLM workflows via RAG can enhance LLM accuracy for clinical applications, particularly in scenarios common to interventional cardiology. These results support the potential for LLMs, when enhanced with domain-specific knowledge, to optimize clinical decision making and increase alignment with guidelines.