Large language models for patient education prior to interventional radiology procedures: a comparative study

Area of Science:

• Interventional Radiology
• Artificial Intelligence in Medicine
• Medical Education

Background:

• Patient comprehension of complex interventional radiology procedures is crucial for informed consent and adherence.
• Large Language Models (LLMs) offer a potential tool to augment patient education and streamline clinical workflows.
• Evaluating LLM performance in answering patient-specific questions related to procedures like TAPE, CT-HDR brachytherapy, and BEST is essential.

Purpose of the Study:

• To assess the accuracy and suitability of four LLMs (ChatGPT-4o, DeepSeek-V3, OpenBioLLM-8b, BioMistral-7b) in responding to patient queries about transarterial periarticular embolization (TAPE), CT-guided high-dose-rate brachytherapy (CT-HDR), and bleomycin electrosclerotherapy (BEST).
• To determine the potential of these LLMs to improve patient understanding and clinical efficiency in interventional radiology settings.
• To identify the risks and limitations associated with using LLMs for patient education in these specialized procedures.

Main Methods:

• A total of 105 patient-related questions (35 TAPE, 34 CT-HDR, 36 BEST) were input into the selected LLMs.
• Responses were evaluated by two board-certified radiologists using a 5-point Likert scale for accuracy and suitability for patient education.
• Statistical analysis was performed to compare LLM performance across different question categories and procedures.

Main Results:

• DeepSeek-V3 and ChatGPT-4o demonstrated high performance, achieving comparable scores for TAPE-related questions and excelling in BEST and CT-HDR brachytherapy.
• OpenBioLLM-8b and BioMistral-7b showed significantly lower performance across all evaluated procedures.
• No statistically significant differences in performance were observed across procedures for the Preparation/Planning question category.

Conclusions:

• DeepSeek-V3 and ChatGPT-4o show significant potential to enhance patient education in interventional radiology, particularly for complex procedures like TAPE, BEST, and CT-HDR brachytherapy.
• The current limitations and inaccuracies of OpenBioLLM-8b and BioMistral-7b highlight that LLMs cannot fully replace comprehensive clinical consultations.
• Further validation through patient feedback and clinical implementation studies is recommended to confirm the utility and safety of LLMs in this domain.