Federated knowledge retrieval elevates large language model performance on biomedical benchmarks

Area of Science:

• Biomedical research
• Artificial intelligence
• Knowledge representation

Background:

• Large language models (LLMs) offer advanced natural language processing for biomedical research.
• LLMs can produce factual inaccuracies (hallucinations) due to reliance on implicit data.
• These inaccuracies pose risks in critical biomedical applications.

Purpose of the Study:

• To develop a framework that improves LLM accuracy in biomedical research.
• To integrate explicit mechanistic evidence with LLM reasoning.
• To enhance factual accuracy and reduce hallucinations in LLM outputs.

Main Methods:

• Developed BTE-RAG, a retrieval-augmented generation framework.
• Integrated LLM reasoning with explicit evidence from BioThings Explorer (API federation).
• Evaluated BTE-RAG against LLM-only methods on three custom benchmark datasets (gene mechanisms, metabolite effects, drug-biological processes).

Main Results:

• BTE-RAG significantly improved accuracy on gene-centric tasks (e.g., GPT-4o accuracy increased from 69.8% to 78.6%).
• Enhanced response quality for metabolite effects (e.g., 82% increase in high cosine similarity for GPT-4o mini).
• Improved answer concordance for drug-biological process relationships and outperformed alternative models on gene-disease association benchmarks.

Conclusions:

• Federated knowledge retrieval via BTE-RAG offers transparent accuracy improvements for LLMs.
• BTE-RAG is a practical tool for mechanistic exploration in biomedical research.
• The framework supports translational biomedical research by enhancing LLM reliability.