Federated Knowledge Retrieval Elevates Large Language Model Performance on Biomedical Benchmarks

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Medicine
• Computational Biology

Background:

• Large language models (LLMs) show promise in biomedical natural language processing.
• LLMs often produce factual inaccuracies (hallucinations) due to reliance on statistical patterns.
• These inaccuracies pose risks in critical biomedical applications.

Purpose of the Study:

• To develop and evaluate a novel retrieval-augmented generation framework, BTE-RAG.
• To enhance the accuracy and reliability of LLMs in biomedical research.
• To leverage explicit mechanistic evidence for improved LLM performance.

Main Methods:

• Developed BTE-RAG, integrating LLMs with BioThings Explorer's API federation.
• Created three benchmark datasets from DrugMechDB for gene-centric mechanisms, metabolite effects, and drug-biological process relationships.
• Systematically evaluated BTE-RAG against LLM-only approaches using GPT-4o and GPT-4o mini.

Main Results:

• BTE-RAG improved gene-centric accuracy from 51% to 75.8% (GPT-4o mini) and 69.8% to 78.6% (GPT-4o).
• Metabolite effect question similarity scores (>=0.90) increased by 82% (GPT-4o mini) and 77% (GPT-4o).
• Drug-biological process concordance improved, with a >10% increase in high-agreement answers for GPT-4o.

Conclusions:

• Federated knowledge retrieval enhances LLM accuracy in biomedical contexts.
• BTE-RAG offers transparent improvements for mechanistic exploration.
• BTE-RAG is a practical tool for translational biomedical research.