Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Informatics

Background:

• Large language models (LLMs) show promise in interventional cardiology but are limited by factual inaccuracies (hallucinations).
• Ensuring clinical utility requires improving the reliability and accuracy of LLM outputs in healthcare settings.

Purpose of the Study:

• To evaluate the impact of retrieval-augmented generation (RAG) on the accuracy of LLMs answering questions based on acute coronary syndrome guidelines.
• To compare the performance of different LLMs, with and without RAG, against established clinical guidelines.

Main Methods:

• Three LLMs (ChatGPT-4o, DeepSeek R1, Med-PaLM 2) were assessed using 38 guideline-based cardiology questions.
• ChatGPT-4o and DeepSeek R1 were tested with and without RAG; Med-PaLM 2 was tested without RAG.
• Model responses were compared to guideline recommendations using an AI-powered similarity scoring tool.

Main Results:

• DeepSeek R1 with RAG demonstrated the highest accuracy at 94.7%, followed by ChatGPT-4o with RAG at 92.1%.
• RAG significantly improved ChatGPT-4o's accuracy from 71.1% to 92.1% (P = 0.017).
• Without RAG, DeepSeek R1 (78.9%) outperformed ChatGPT-4o (71.1%) and Med-PaLM 2 (68.4%).

Conclusions:

• Integrating guideline content into LLM workflows using RAG enhances accuracy for clinical applications, especially in interventional cardiology.
• Domain-specific knowledge augmentation via RAG supports optimized clinical decision-making and adherence to medical guidelines.
• LLMs, when enhanced with RAG, hold significant potential for improving healthcare practices and patient outcomes.