RadioRAG: Online Retrieval-Augmented Generation for Radiology Question Answering

Area of Science:

• Medical Informatics
• Artificial Intelligence in Radiology
• Natural Language Processing

Background:

• Large Language Models (LLMs) show promise for medical question answering.
• LLMs can generate inaccurate information (hallucinations) without access to real-time data.
• Retrieval-augmented generation (RAG) integrates external knowledge to improve LLM factuality.

Purpose of the Study:

• To assess the diagnostic accuracy of LLMs in radiology question answering.
• To evaluate the impact of Retrieval-Augmented Generation (RAG) on LLM performance.
• To compare LLM performance with and without RAG against human expert accuracy.

Main Methods:

• Developed RadioRAG, a framework for real-time data retrieval from radiologic sources.
• Evaluated multiple LLMs (GPT-3.5-turbo, GPT-4, Mistral, Llama3) with and without RadioRAG.
• Used 80 RSNA Case Collection questions and 24 expert-curated questions for assessment.

Main Results:

• RadioRAG improved accuracy for GPT-3.5-turbo (74% vs 66%) and Mixtral 8x7B (76% vs 65%) on the RSNA-RadioQA dataset.
• LLM accuracy with RAG surpassed human expert performance (63%) for some models.
• RadioRAG reduced hallucinations across all tested LLMs (6%-25% rate) but increased response time fourfold.

Conclusions:

• RadioRAG demonstrates potential to enhance LLM accuracy and factuality in radiology.
• Integrating real-time, domain-specific data via RAG is crucial for reliable AI in medical QA.
• Further research can optimize RAG for faster, more accurate AI-assisted radiology.