Enhancing Large Language Models with Retrieval-Augmented Generation: A Radiology-Specific Approach.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Radiology

Background:

• Large language models (LLMs) show potential in medicine but require up-to-date, citable information.
• Retrieval-augmented generation (RAG) offers a method to enhance LLMs with external knowledge bases in real-time.
• RAG may improve LLM performance and clinical applicability in radiology without costly model fine-tuning.

Purpose of the Study:

• To develop and evaluate a radiology-specific RAG system for enhancing LLM performance on radiology knowledge tasks.
• To compare the performance of multiple LLMs with and without RAG on a radiology examination.
• To assess the ability of RAG systems to retrieve and cite relevant domain-specific information.

Main Methods:

• A radiology-specific RAG system was created using a vector database of 3689 RadioGraphics articles (1999-2023).
• Five LLMs (GPT-4, Command R+, Claude Opus, Mixtral, Gemini 1.5 Pro) were tested with and without RAG on a 192-question radiology exam.
• Performance was evaluated based on examination scores and the retrieval/citation of relevant references.

Main Results:

• RAG significantly improved scores for GPT-4 (81.2% vs 75.5%) and Command R+ (70.3% vs 62.0%).
• RAG systems outperformed pure LLMs on a subset of RadioGraphics-sourced questions (85% vs 76%).
• RAG systems successfully retrieved and cited relevant references from the RadioGraphics database.

Conclusions:

• Retrieval-augmented generation (RAG) is a promising strategy for enhancing LLM capabilities in radiology.
• RAG systems provide transparent, domain-specific information retrieval, improving accuracy on knowledge-based tasks.
• This approach supports the clinical applicability of LLMs in radiology by ensuring access to current, citable knowledge.