Related Experiment Video
Updated: May 15, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Evaluating large language models for evidence-based clinical question answering
1Department of Biostatistics, Johns Hopkins University, Baltimore, MD 21205, USA.
Patterns (New York, N.Y.)
|May 14, 2026
Summary
Large language models (LLMs) show promise in medicine, but their reliability needs testing. Structured guidelines improve LLM accuracy, while retrieval augmentation enhances performance for evidence-based medicine.
Area of Science:
- Artificial Intelligence
- Medical Informatics
- Clinical Decision Support
Background:
- Large language models (LLMs) demonstrate potential for clinical applications.
- Rigorous evaluation is crucial to ensure the reliability of LLMs in evidence-based medicine.
- Existing benchmarks may not fully capture the complexities of clinical data.
Purpose of the Study:
- To evaluate the performance of leading LLMs (GPT-5, GPT-4o-mini, Claude 4, DeepSeek-v3) on a diverse clinical question-answering benchmark.
- To identify factors influencing LLM accuracy, including source type, citation count, and geographic origin.
- To assess the impact of retrieval-augmented generation (RAG) on LLM performance in a clinical context.
Main Methods:
- Curated a benchmark of over 20,000 question-answering pairs from systematic reviews and clinical guidelines.
- Assessed the accuracy of four LLMs across different source types: structured guidelines, narrative sources, and systematic reviews.
- Investigated correlations between accuracy and source characteristics (citation count, publication year, geographic origin) and evaluated RAG strategies using PubMed retrieval.
Main Results:
- LLM accuracy varied by source type: highest with structured guidelines (90%), moderate with narrative sources (70%), and lowest with systematic reviews (50%-60%).
- Higher citation counts correlated with increased accuracy; moderate geographic variation was observed.
- Retrieval-augmented generation, using top PubMed articles, improved accuracy by 23%, a consistent finding across models.
Conclusions:
- LLM performance is significantly influenced by the clarity and structure of source materials.
- Targeted retrieval strategies, such as RAG, are effective in enhancing LLM factual alignment.
- Stratified evaluation and optimized retrieval are essential for deploying LLMs reliably in clinical decision-making.
Related Concept Videos
01:16Clinical Trials
Clinical trials are prospective experimental studies conducted on humans to determine the safety and efficacy of treatments, drugs, diet methods, and medical devices. Using statistics in clinical trials enables researchers to derive reasonable and accurate conclusions from the collected data, allowing them to make wise decisions in uncertain situations. In medical research, statistical methods are crucial for preventing errors and bias.
There are four phases in a clinical trial. A phase one...
There are four phases in a clinical trial. A phase one...
01:11Clinical Trials: Overview
Clinical development focuses on how the drug will interact with the human body and encompasses four key phases of clinical trials, each serving a specific purpose in assessing the safety and effectiveness of new drugs. These phases overlap and build upon one another. Phase I involves a small group of healthy volunteers (typically 20-80 individuals) or, in cases where significant toxicity is expected, patients with the targeted disease, such as cancer or AIDS. The volunteers are tested for...