Development and Evaluation of a Retrieval-Augmented Generation System for Radiology Guidelines

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Medical Decision Support

Background:

• Large language models (LLMs) show promise for processing medical information but lack reliability and up-to-date knowledge for clinical decision-making.
• Time-sensitive fields like radiology require rapid access to accurate, current guidelines for optimal patient care.
• Manual guideline lookup is inefficient, hindering timely clinical application.

Purpose of the Study:

• To develop and evaluate a retrieval-augmented generation (RAG) system for grounding LLM responses in European Society of Urogenital Radiology (ESUR) guidelines.
• To improve the accuracy, completeness, and clinical usefulness of LLM-generated information based on authoritative medical guidelines.

Main Methods:

• Developed a RAG system using headline-based chunking and the all-mpnet-base-v2 embedding model for information extraction and retrieval.
• Evaluated the RAG system against a standalone LLM and an enhanced iterative RAG system using 79 queries.
• Assessed performance based on retrieval accuracy, factual correctness, completeness, and clinical usefulness.

Main Results:

• Both RAG systems significantly outperformed the standalone LLM across all evaluated metrics.
• The enhanced RAG system achieved the highest scores for factual accuracy (0.89), completeness (4.20/5), and usefulness (3.99/5).
• Minor, non-significant improvements were observed for the enhanced RAG over the standard version.

Conclusions:

• Grounded RAG systems demonstrate significant potential to enhance clinical guideline accessibility and reliability in medical settings.
• Further validation and refinement, including domain-specific embeddings and query expansion, are needed for clinical deployment.
• The study highlights the importance of grounding LLMs in authoritative sources to mitigate unreliability and hallucinations in clinical applications.