Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Large Language Models (LLMs) show promise in medicine but often generate unsupported or hallucinated information.
• Retrieval Augment Generation (RAG) is a technique to improve LLM factuality, yet its application in specialized medical domains requires thorough evaluation.

Purpose of the Study:

• To develop and evaluate a RAG pipeline using ophthalmology-specific documents to augment LLMs for medical question answering.
• To systematically assess the impact of RAG on the factuality, evidence selection, attribution, accuracy, and completeness of LLM responses.

Main Methods:

• A RAG pipeline was created, incorporating approximately 70,000 ophthalmology documents.
• LLM responses to 100 long-form consumer health questions were evaluated by 10 healthcare professionals, comparing RAG-augmented and non-augmented models.
• Evaluation metrics included evidence factuality, ranking, attribution, and answer accuracy/completeness.

Main Results:

• LLMs without RAG produced responses with 45.3% hallucinated and 34.1% erroneous references.
• RAG significantly improved response accuracy to 54.5% and reduced errors (18.8% minor hallucinations, 26.7% errors).
• RAG improved evidence attribution but showed slight decreases in accuracy and completeness; LLMs did not always utilize top-ranked retrieved documents.

Conclusions:

• RAG substantially enhances the reliability of LLM-generated medical information, reducing factual errors and hallucinations.
• Challenges persist in RAG's document retrieval and selection processes, impacting LLM performance in complex tasks like long-form question answering.
• Further advancements in domain-specific RAG and LLM techniques are necessary to overcome current limitations in evidence utilization and response quality.