Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in medical question answering but struggle with knowledge limitations and hallucinations.
• Retrieval-Augmented Generation (RAG) improves LLM accuracy by integrating external knowledge but can falter in complex, multi-step queries.

Purpose of the Study:

• To introduce iterative RAG for medicine (i-MedRAG), a novel framework enabling LLMs to conduct multi-turn information retrieval for enhanced medical question answering.
• To evaluate the performance of i-MedRAG against traditional RAG and other advanced methods on challenging medical and general knowledge benchmarks.

Main Methods:

• Developed i-MedRAG, where LLMs generate follow-up queries based on prior retrieval attempts, creating iterative information-seeking loops.
• Tested i-MedRAG with various LLMs on clinical vignettes from the United States Medical Licensing Examination (USMLE) and the Massive Multitask Language Understanding (MMLU) dataset.
• Analyzed the impact of iteration depth and query count on i-MedRAG's performance and scalability.

Main Results:

• i-MedRAG significantly improved LLM performance on complex USMLE and MMLU questions compared to vanilla RAG.
• Zero-shot i-MedRAG on GPT-3.5 surpassed existing prompt engineering and fine-tuning techniques, achieving 69.68% accuracy on the MedQA dataset.
• Case studies demonstrated i-MedRAG's ability to construct reasoning chains through flexible follow-up queries for in-depth medical question analysis.

Conclusions:

• i-MedRAG offers a robust solution for enhancing LLM-based medical question answering, particularly for complex queries requiring multi-step reasoning.
• The iterative nature of i-MedRAG allows for dynamic knowledge acquisition and refinement, addressing limitations of static RAG systems.
• This study pioneers the integration of follow-up queries within a medical RAG framework, paving the way for more sophisticated AI-driven medical information systems.