Optimizing Retrieval-Augmented Generation (RAG) in clinical medicine: methods and performance evaluation

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Natural Language Processing

Background:

• Large Language Models (LLMs) show promise for medical knowledge retrieval.
• Optimizing Retrieval-Augmented Generation (RAG) architecture is crucial for accurate medical applications.
• Sleep medicine presents unique challenges for AI due to specialized terminology and complex case data.

Purpose of the Study:

• To evaluate the impact of RAG architecture components on LLM performance in sleep medicine.
• To assess how corpus structure, retrieval strategy, and pipeline complexity influence medical problem-solving.
• To benchmark open-source LLMs within a sleep medicine knowledge domain.

Main Methods:

• Benchmarked four LLMs (Llama-3-8B, Llama-3-70B, Qwen 2.5-14B, Qwen 2.5-235B) using a sleep medicine textbook knowledge base.
• Compared performance across corpus structures (raw text vs. table-of-contents aligned), retrieval strategies (dense vs. hybrid sparse-dense), and pipeline complexity (baseline vs. augmented).
• Evaluated using multiple-choice question (MCQ) accuracy and clinical case vignette diagnostic ranking.

Main Results:

• RAG enhanced MCQ accuracy for all tested LLMs, with gains up to 25.5% (Llama-8B: 61.8% to 72.4%, Qwen-235B: 87.3%).
• Structured corpora improved primary diagnosis accuracy by 6.1% on average, with Qwen-235B showing a peak 10.2% increase.
• Hybrid retrieval and structured data corrected context noise issues seen with dense retrieval on raw text, particularly for smaller models.

Conclusions:

• RAG effectiveness is a balance between LLM size and data structure; smaller models benefit significantly from structured data.
• Hybrid retrieval is essential for precision with specialized medical terms in sleep medicine.
• A structured corpus with a baseline hybrid pipeline offers optimal stability and speed for clinical AI deployment.