Benchmarking large language models for question answering on German clinical practice guidelines

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Evidence-Based Medicine

Background:

• Clinical practice guidelines (CPGs) are crucial for evidence-based medicine but face inconsistent implementation.
• Large language models (LLMs) show potential for medical question answering, but their use with German CPGs is underexplored.
• A dedicated benchmark is needed to evaluate LLM performance on German CPGs.

Purpose of the Study:

• To develop and validate a benchmark dataset for evaluating guideline-based question answering using German clinical practice guidelines.
• To assess the performance of different LLMs and retrieval configurations within a Retrieval-Augmented Generation (RAG) framework.
• To investigate the potential of LLMs for improving the accessibility and application of CPGs in the German healthcare system.

Main Methods:

• Development of cpgQA-DE, an expert-validated benchmark with 200 multiple-choice questions from 10 German CPGs across five specialties.
• Inclusion of case-based and knowledge-based questions with metadata on source, specialty, relevance, and difficulty.
• Evaluation of LLM performance using a RAG pipeline with various model-retriever combinations on the cpgQA-DE dataset.

Main Results:

• Retrieval-Augmented Generation (RAG) significantly enhanced accuracy for all tested LLMs.
• The top configuration (GPT-5 with multilingual-e5-large retriever) achieved 95% accuracy.
• An open-weight model (gpt-oss-120b) demonstrated 90% accuracy when integrated with RAG.

Conclusions:

• The cpgQA-DE benchmark facilitates reproducible offline evaluation of guideline-aware QA systems in German healthcare.
• RAG integration boosts LLM performance, supporting the use of external, quality-assured knowledge for clinical decision support.
• High performance of open-weight models indicates potential for privacy-preserving on-premises deployment in clinical settings.