Does Domain-Specific Retrieval Augmented Generation Help LLMs Answer Consumer Health Questions?

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) demonstrate strong performance on medical tasks.
• The efficacy of retrieval-augmented generation (RAG) for consumer health questions remains unclear.
• Evaluating RAG's impact on LLM-based medical question-answering is crucial.

Purpose of the Study:

• To systematically compare vanilla LLMs with RAG-enhanced LLMs for consumer health question-answering.
• To assess performance using automated metrics, LLM-based evaluation, and clinical validation.
• To identify challenges and requirements for effective RAG implementation in medical AI.

Main Methods:

• Four open-source LLMs were evaluated in both vanilla and RAG configurations.
• The NIDDK portion of the MedQuAD dataset was utilized for systematic evaluation.
• Performance was measured using quantitative metrics (BLEU, ROUGE, BERTScore), LLM-based assessment, and clinical validation.

Main Results:

• Vanilla LLM approaches consistently outperformed RAG variants across all evaluation metrics.
• Low retrieval performance (Precision@5 = 0.15) indicates significant challenges in current RAG systems for medical queries.
• RAG showed competitive results only in specific domains like scientific consensus and harm reduction.

Conclusions:

• Simple retrieval and prompt engineering are insufficient for effective RAG in consumer health question-answering.
• Developing medical-specific RAG infrastructure is necessary to improve medical AI systems.
• Further research into sophisticated RAG approaches is required for reliable medical question-answering.