Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Informatics

Background:

• Large Language Models (LLMs) show advanced performance on medical licensing exams.
• The translation of LLM capabilities from knowledge-based testing to clinical practice is not well understood.
• A gap exists in evaluating AI readiness for real-world clinical deployment.

Purpose of the Study:

• To systematically review and analyze medical LLM benchmarks.
• To categorize evaluation paradigms and identify limitations in current assessment methodologies.
• To quantify the performance gap between knowledge-based and practice-based LLM evaluations.

Main Methods:

• Systematic review registered on PROSPERO (CRD420251139729).
• Searched MEDLINE/PubMed, Embase/Ovid, Cochrane Library, and arXiv up to August 31, 2025.
• Included studies evaluating clinical medicine benchmarks in LLMs; excluded non-medical domains or unvalidated benchmarks.
• Assessed methodological quality using the Mixed Methods Appraisal Tool; conducted narrative synthesis due to heterogeneity.

Main Results:

• Identified 39 medical LLM benchmarks (21 knowledge-based, 15 practice-based, 3 hybrid).
• LLMs achieve 84%-90% accuracy on knowledge-based exams but only 45%-69% on practice-based assessments.
• Performance varies significantly by task: factual retrieval (85%-93%), clinical reasoning (50%-60%), diagnosis (45%-55%), and safety (40%-50%).
• 26% of benchmarks had insufficient methodological reporting.

Conclusions:

• A significant "knowledge-practice gap" exists in medical AI, with high exam scores not correlating to clinical competence.
• Examination scores are insufficient and misleading indicators of clinical readiness for LLMs.
• Autonomous deployment is not currently justified; practice-oriented validation and human oversight are crucial for patient safety.