Agentic memory-augmented retrieval and evidence grounding for medical question-answering tasks

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in medical question-answering.
• Standalone LLMs often struggle with complex, multi-step medical reasoning.
• The integration of external tools and evidence grounding is crucial for enhancing AI reliability in medicine.

Purpose of the Study:

• To evaluate if a tool-using agent-based system built on LLMs outperforms standalone LLMs in medical question-answering.
• To develop and assess an open-source LLM-based agentic system for dynamic, multi-step medical reasoning.

Main Methods:

• Developed a unified, open-source LLM-based agentic system with document retrieval, reranking, evidence grounding, and diagnosis generation.
• Implemented a retrieval-augmented generation pipeline and a cache-and-prune memory bank for efficient long-context inference.
• The system autonomously invoked specialized tools, bypassing manual prompt engineering.

Main Results:

• The agentic system achieved high accuracies on medical question-answering benchmarks: 82.98% on USMLE Step 1 and 86.24% on Step 2.
• Performance surpassed GPT-4 on USMLE Steps 1 and 2, while closely matching on Step 3.
• The system demonstrated superior or comparable performance to state-of-the-art models in both multiple-choice and open-ended formats.

Conclusions:

• Combining tool-augmented and evidence-grounded reasoning strategies is valuable for developing reliable and scalable medical AI systems.
• Agent-based LLM systems offer a promising approach to enhance AI capabilities in complex medical reasoning tasks.
• The developed system provides an open-source framework for advancing AI in medical question answering.