Retrieval-augmented generation elevates local LLM quality in radiology contrast media consultation

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Radiology

Background:

• Large language models (LLMs) show promise in healthcare but face barriers like privacy concerns and limited medical training.
• Clinical adoption of LLMs is hindered by data security and the need for specialized medical knowledge.

Purpose of the Study:

• To evaluate if retrieval-augmented generation (RAG) can enhance locally deployable LLMs for radiology contrast media consultations.
• To assess the performance of a RAG-enhanced LLM against cloud-based models in a clinical context.

Main Methods:

• Compared Llama 3.2-11B (baseline and RAG-enhanced) with GPT-4o mini, Gemini 2.0 Flash, and Claude 3.5 Haiku on 100 synthetic contrast media consultations.
• Utilized a blinded radiologist's ranking and LLM-based judges scoring accuracy, safety, structure, tone, applicability, and latency.

Main Results:

• RAG eliminated hallucinations (0% vs 8%) and improved mean rank by 1.3 (p < 0.001).
• The RAG-enhanced model was faster (2.6s vs 4.9-7.3s) and preferred by LLM judges over GPT-4o mini.
• Performance gaps with cloud models persisted, though a radiologist ranked GPT-4o mini higher.

Conclusions:

• RAG offers significant improvements for local clinical LLMs, enhancing accuracy and reducing hallucinations.
• RAG maintains the privacy benefits of on-premise deployment for healthcare AI.
• Further research is needed to bridge performance gaps with leading cloud-based models.