MedDiscover: A Domain-Specific Retrieval-Augmented Generation Framework for Evidence-Grounded Knowledge Extraction in...

Area of Science:

• Biomedical informatics
• Natural Language Processing
• Metabolomics

Background:

• Retrieval-augmented generation (RAG) enhances biomedical question answering.
• Domain-specific retriever evaluation needs transparency and reproducibility.
• Metabolomics and metabolic disorders literature present unique challenges for RAG.

Purpose of the Study:

• Introduce MedDiscover, an open-source RAG implementation and benchmark for metabolomics.
• Provide a reproducible evaluation framework for domain-specific RAG systems.
• Compare the performance of different retrieval methods (Ada, MedCPT) in a RAG context.

Main Methods:

• Developed MedDiscover with a two-tier dataset: Gold (expert-curated) and Silver (ICD-coded).
• Created expert-curated and synthetic question-answer pairs with retrievability metadata.
• Evaluated RAG performance on grounding, faithfulness, context recall/precision, answer relevancy, and correctness.

Main Results:

• Retrieval augmentation improved grounding on the Gold set compared to a non-RAG baseline.
• MedCPT and Ada demonstrated comparable retrieval-centric performance on the Silver benchmark.
• MedCPT significantly outperformed Ada in answer correctness on the Silver benchmark.

Conclusions:

• MedDiscover provides a robust benchmark for evaluating RAG in specialized biomedical domains.
• The study highlights MedCPT's superior performance in generating correct answers for metabolomics queries.
• Open-sourced code, scripts, and document lists promote reproducible research in biomedical RAG.