Retrieval-Augmented Language Models for Patient-Centered Periprocedural Anticoagulation in Interventional Radiology | JoVE Visualize

Area of Science:

Artificial Intelligence in Medicine
Clinical Decision Support Systems
Medical Informatics

Background:

Accurate anticoagulation (AC) management is critical in Interventional Radiology (IR) to prevent thromboembolic events and bleeding complications.
Current AC guideline adherence can be challenging due to complex patient factors and evolving recommendations.
Large Language Models (LLMs) offer potential for processing complex medical information and generating recommendations.

Purpose of the Study:

To evaluate the performance of state-of-the-art LLMs integrated with Retrieval-Augmented Generation (RAG) frameworks in providing guideline-concordant AC recommendations for IR procedures.
To compare the accuracy of different LLMs (Claude-3.5-Sonnet, GPT-4o, LLaMA-3.1, Mistral-Large-2411, Qwen-2.5) in managing AC parameters like withholding, reinitiation, platelet, and INR thresholds.
To assess the impact of RAG integration on LLM performance for AC recommendations.

Main Methods:

A dataset of 394 Interventional Radiology (IR) procedure-anticoagulation (AC) pairs was extracted from the MIMIC-IV database.
Five LLMs were deployed within a RAG system utilizing LlamaIndex to index Society of Interventional Radiology (SIR) guidelines.
Model outputs were evaluated for guideline concordance on AC withholding, timing, and threshold parameters, with scores assigned by a human assessor.

Main Results:

Claude-3.5-Sonnet achieved the highest overall guideline concordance (1.51/2), with RAG integration significantly improving its performance across most categories.
Claude, GPT-4o, and Mistral-Large-2411 demonstrated similar, strong performance in AC withholding decisions.
Claude and GPT-4o showed comparable accuracy in withholding and reinitiation timeframes, while Mistral led in platelet threshold concordance and Claude in INR recommendations. LLaMA-3.1 and Qwen-2.5 significantly underperformed.

Conclusions:

LLMs, particularly Claude, GPT-4o, and Mistral, when integrated with RAG, show significant potential as safe and efficient clinical decision-support tools for generating guideline-concordant AC recommendations in IR.
These AI-driven systems can be customized to provide patient-specific AC advice, supporting clinicians in complex scenarios.
Further development and validation are warranted to fully integrate these tools into clinical workflows for optimizing patient care in IR.