Evaluating Retrieval-Augmented Generation-Large Language Models for Infective Endocarditis Prophylaxis: Clinical...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Informatics

Background:

• Large language models (LLMs) are increasingly used in healthcare.
• Retrieval-augmented generation (RAG) enhances LLMs by grounding responses in specific, current data.
• Limitations of standard LLMs necessitate methods like RAG for reliable medical applications.

Purpose of the Study:

• To evaluate RAG-augmented LLMs for infective endocarditis (IE) prophylaxis recommendations in dental procedures.
• To compare the performance of RAG-LLMs against non-RAG LLMs using a standardized question set.
• To explore the utility of LLMs as a clinical decision support tool through a pilot study with dental students.

Main Methods:

• Ten RAG-integrated LLMs were tested using the 2021 American Heart Association IE guideline.
• A consistent IE prophylaxis question set from prior research was utilized for comparability.
• Performance was assessed with and without a preprompt, and a pilot study evaluated LLM assistance for dental students.

Main Results:

• Grok 3 beta achieved 90.0% accuracy with preprompting; DeepSeek Reasoner had the highest accuracy (83.6%) without preprompting.
• Preprompting generally improved LLM accuracy, though RAG's impact varied by model.
• The pilot study indicated mixed results for LLM assistance on accuracy and a significant increase in task time for students.

Conclusions:

• RAG and prompt engineering can improve LLM performance for clinical decision support.
• Current LLMs with RAG offer rapid information access but require critical evaluation due to potential inaccuracies.
• Clinicians and students must exercise digital literacy and maintain professional judgment when using these AI tools.