Performance of Large Language Models on the Acute Coronary Syndrome Guidelines Using Retrieval-Augmented Generation
These videos have been matched automatically. Contact us if they are not relevant.
These videos have been matched automatically. Contact us if they are not relevant.
View abstract on PubMed
Summary
This summary is machine-generated.Retrieval-augmented generation (RAG) significantly improves large language model (LLM) accuracy in cardiology guidelines. DeepSeek R1 with RAG achieved 94.7% accuracy, enhancing clinical decision-making potential.
Area Of Science
- Artificial Intelligence in Medicine
- Clinical Decision Support Systems
- Medical Informatics
Background
- Large language models (LLMs) show promise in interventional cardiology but are limited by factual inaccuracies (hallucinations).
- Ensuring clinical utility requires improving the reliability and accuracy of LLM outputs in healthcare settings.
Purpose Of The Study
- To evaluate the impact of retrieval-augmented generation (RAG) on the accuracy of LLMs answering questions based on acute coronary syndrome guidelines.
- To compare the performance of different LLMs, with and without RAG, against established clinical guidelines.
Main Methods
- Three LLMs (ChatGPT-4o, DeepSeek R1, Med-PaLM 2) were assessed using 38 guideline-based cardiology questions.
- ChatGPT-4o and DeepSeek R1 were tested with and without RAG; Med-PaLM 2 was tested without RAG.
- Model responses were compared to guideline recommendations using an AI-powered similarity scoring tool.
Main Results
- DeepSeek R1 with RAG demonstrated the highest accuracy at 94.7%, followed by ChatGPT-4o with RAG at 92.1%.
- RAG significantly improved ChatGPT-4o's accuracy from 71.1% to 92.1% (P = 0.017).
- Without RAG, DeepSeek R1 (78.9%) outperformed ChatGPT-4o (71.1%) and Med-PaLM 2 (68.4%).
Conclusions
- Integrating guideline content into LLM workflows using RAG enhances accuracy for clinical applications, especially in interventional cardiology.
- Domain-specific knowledge augmentation via RAG supports optimized clinical decision-making and adherence to medical guidelines.
- LLMs, when enhanced with RAG, hold significant potential for improving healthcare practices and patient outcomes.
These videos have been matched automatically. Contact us if they are not relevant.
These videos have been matched automatically. Contact us if they are not relevant.
Related Concept Videos
Diagnosing acute coronary syndrome or ACS begins with a thorough patient history. Notable symptoms include central, crushing chest pain radiating to the left arm, neck, jaw, or back, along with shortness of breath, sweating (diaphoresis), nausea, vomiting, dizziness, and palpitations.It is crucial to note any history of cardiac illnesses and assess risk factors, including age, gender, smoking, hypertension, diabetes, hyperlipidemia, and a sedentary lifestyle.During physical examination, vital...
Acute Coronary Syndrome (ACS) encompasses a spectrum of heart conditions caused by sudden obstruction of coronary arteries, typically resulting from the rupture of an atherosclerotic plaque and subsequent thrombus (blood clot) formation. This obstruction can lead to partial or complete blockage of blood flow, causing varying degrees of myocardial ischemia or infarction.ACS includes the following clinical entities:Unstable Angina (UA)Non-ST-Elevation Myocardial Infarction (NSTEMI)ST-Elevation...
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...