Retrieval-Augmented Language Models for Patient-Centered Periprocedural Anticoagulation in Interventional Radiology

This study evaluates Large Language Models (LLMs) integrated with Retrieval-Augmented Generation (RAG) frameworks for generating accurate, guideline-concordant anticoagulation (AC) recommendations in Interventional Radiology (IR) MATERIALS AND METHODS: 394 IR procedure-AC pairs were extracted from the Medical Information Mart for Intensive Care-IV (MIMIC-IV) database. Five state-of-the-art LLMs, Claude-3.5-Sonnet, GPT-4o, LLaMA-3.1, Mistral-Large-2411, and Qwen-2.5, were deployed within a RAG system built using LlamaIndex, which indexed the Society of Interventional Radiology (SIR) guidelines. Claude was evaluated without RAG integration (Base Claude). Models were evaluated on withholding AC, withholding and reinitiation timeframes, and platelet and International Normalized Ratio (INR) thresholds. A human assessor scored outputs for guideline concordance (2 = complete, 1 = partial, 0 = none), which were subsequently averaged.

Claude achieved the highest overall guideline concordance (1.51/2). Claude, GPT-4o, and Mistral performed similarly in withholding decisions (1.88 [95% CI 1.83, 1.93], 1.86 [1.81, 1.91], and 1.88 [1.84, 1.93]). Claude and GPT-4o showed comparable accuracy in both withholding (1.56 [1.44, 1.67] vs. 1.55 [1.44, 1.67]; p=0.871) and reinitiation timeframes (1.34 [1.20, 1.47] vs. 1.37 [1.26, 1.49]; p=0.920). Mistral achieved the highest platelet threshold concordance (1.52 [1.45, 1.59]), while Claude led in International Normalized Ratio (INR) recommendations (1.37 [1.32, 1.42]). LLaMA (0.76) and Qwen (0.52) significantly underperformed across all categories (p<0.001). RAG integration significantly improved Claude's performance across all categories, except platelet thresholds.

LLMs integrated with RAG systems, especially Claude, GPT-4o, and Mistral, demonstrate potential for delivering guideline-concordant, patient-specific AC recommendations in IR, supporting their role as safe, efficient, and customizable clinical decision-support tools.