Almanac - Retrieval-Augmented Language Models for Clinical Medicine

Area of Science:

• Artificial Intelligence
• Clinical Medicine
• Natural Language Processing

Background:

• Large language models (LLMs) exhibit zero-shot capabilities for various natural language tasks.
• Despite potential in clinical medicine, LLM adoption is hindered by factual inaccuracies and safety concerns.

Purpose of the Study:

• To evaluate Almanac, an LLM framework with retrieval from curated medical resources, for medical guideline and treatment recommendations.
• To compare Almanac's performance against standard LLMs (ChatGPT-4, Bing, Bard) using clinical questions.

Main Methods:

• A panel of eight board-certified clinicians and two health care practitioners evaluated LLM responses.
• The evaluation used a novel dataset of 314 clinical questions across nine medical specialties.
• Responses from Almanac, ChatGPT-4, Bing, and Bard were compared.

Main Results:

• Almanac demonstrated significant improvements in factuality and completeness compared to standard LLMs.
• Almanac also showed enhanced user preference and adversarial safety.
• Performance was assessed across nine medical specialties.

Conclusions:

• LLMs accessing domain-specific corpora hold potential for clinical decision-making.
• Rigorous testing of LLMs is crucial before clinical deployment to address limitations.
• Findings supported by the National Institutes of Health, National Heart, Lung, and Blood Institute.