Evaluating large language models for evidence-based clinical question answering.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Clinical Decision Support

Background:

• Large language models (LLMs) demonstrate potential for clinical applications.
• Rigorous evaluation is crucial to ensure the reliability of LLMs in evidence-based medicine.
• Existing benchmarks may not fully capture the complexities of clinical data.

Purpose of the Study:

• To evaluate the performance of leading LLMs (GPT-5, GPT-4o-mini, Claude 4, DeepSeek-v3) on a diverse clinical question-answering benchmark.
• To identify factors influencing LLM accuracy, including source type, citation count, and geographic origin.
• To assess the impact of retrieval-augmented generation (RAG) on LLM performance in a clinical context.

Main Methods:

• Curated a benchmark of over 20,000 question-answering pairs from systematic reviews and clinical guidelines.
• Assessed the accuracy of four LLMs across different source types: structured guidelines, narrative sources, and systematic reviews.
• Investigated correlations between accuracy and source characteristics (citation count, publication year, geographic origin) and evaluated RAG strategies using PubMed retrieval.

Main Results:

• LLM accuracy varied by source type: highest with structured guidelines (90%), moderate with narrative sources (70%), and lowest with systematic reviews (50%-60%).
• Higher citation counts correlated with increased accuracy; moderate geographic variation was observed.
• Retrieval-augmented generation, using top PubMed articles, improved accuracy by 23%, a consistent finding across models.

Conclusions:

• LLM performance is significantly influenced by the clarity and structure of source materials.
• Targeted retrieval strategies, such as RAG, are effective in enhancing LLM factual alignment.
• Stratified evaluation and optimized retrieval are essential for deploying LLMs reliably in clinical decision-making.