Retrieval-Augmented Language Models for Patient-Centered Periprocedural Anticoagulation in Interventional Radiology

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Informatics

Background:

• Accurate anticoagulation (AC) management is critical in Interventional Radiology (IR) to prevent thromboembolic events and bleeding complications.
• Current AC guideline adherence can be challenging due to complex patient factors and evolving recommendations.
• Large Language Models (LLMs) offer potential for processing complex medical information and generating recommendations.

Purpose of the Study:

• To evaluate the performance of state-of-the-art LLMs integrated with Retrieval-Augmented Generation (RAG) frameworks in providing guideline-concordant AC recommendations for IR procedures.
• To compare the accuracy of different LLMs (Claude-3.5-Sonnet, GPT-4o, LLaMA-3.1, Mistral-Large-2411, Qwen-2.5) in managing AC parameters like withholding, reinitiation, platelet, and INR thresholds.
• To assess the impact of RAG integration on LLM performance for AC recommendations.

Main Methods:

• A dataset of 394 Interventional Radiology (IR) procedure-anticoagulation (AC) pairs was extracted from the MIMIC-IV database.
• Five LLMs were deployed within a RAG system utilizing LlamaIndex to index Society of Interventional Radiology (SIR) guidelines.
• Model outputs were evaluated for guideline concordance on AC withholding, timing, and threshold parameters, with scores assigned by a human assessor.

Main Results:

• Claude-3.5-Sonnet achieved the highest overall guideline concordance (1.51/2), with RAG integration significantly improving its performance across most categories.
• Claude, GPT-4o, and Mistral-Large-2411 demonstrated similar, strong performance in AC withholding decisions.
• Claude and GPT-4o showed comparable accuracy in withholding and reinitiation timeframes, while Mistral led in platelet threshold concordance and Claude in INR recommendations. LLaMA-3.1 and Qwen-2.5 significantly underperformed.

Conclusions:

• LLMs, particularly Claude, GPT-4o, and Mistral, when integrated with RAG, show significant potential as safe and efficient clinical decision-support tools for generating guideline-concordant AC recommendations in IR.
• These AI-driven systems can be customized to provide patient-specific AC advice, supporting clinicians in complex scenarios.
• Further development and validation are warranted to fully integrate these tools into clinical workflows for optimizing patient care in IR.