Improving medical reasoning through retrieval and self-reflection with retrieval-augmented large language models.

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Medicine

Background:

• Large language models (LLMs) show promise in biomedical tasks but struggle with domain-specific knowledge.
• Retrieval-augmented generation (RAG) methods improve LLMs by incorporating external documents, yet often face generalization issues in specialized domains.

Purpose of the Study:

• To introduce Self-BioRAG, a novel framework designed for reliable biomedical text generation, explanation, and self-reflection.
• To address the limitations of existing RAG methods in domain-specific applications, particularly in the medical field.

Main Methods:

• Developed Self-BioRAG, a framework incorporating a domain-specific retriever, a specialized document corpus, and tailored instruction sets.
• Trained the model using 84,000 filtered biomedical instruction sets, enabling it to self-assess generated explanations using reflective tokens.
• Utilized three major medical question-answering benchmark datasets for evaluation.

Main Results:

• Self-BioRAG achieved a 7.2% average absolute performance improvement over state-of-the-art open-foundation models (7B or less parameters) on medical question-answering tasks.
• Outperformed standard RAG by an average of 8% in Rouge-1 score on long-form question-answering benchmarks.
• Demonstrated the necessity of domain-specific components for effective adherence to biomedical instructions.

Conclusions:

• Self-BioRAG effectively mimics a medical expert by retrieving relevant information and synthesizing it with encoded knowledge.
• The framework shows significant advancements in biomedical text generation and question-answering capabilities.
• Released data, code, and model weights (7B and 13B) to foster further research and development in the biomedical and clinical domains.