Evaluation Methods for Inference-Time Retrieval-Augmented and Graph Retrieval-Augmented Large Language Models in...

Area of Science:

• * Artificial Intelligence in Healthcare
• * Natural Language Processing
• * Clinical Informatics

Background:

• * Retrieval augmentation is crucial for traceable and verifiable large language model (LLM) applications in healthcare.
• * Current evaluation practices for text-based retrieval-augmented generation (RAG) and graph-structured RAG (GraphRAG) are heterogeneous, impeding cross-study comparisons and clinical readiness assessments.

Purpose of the Study:

• * To map and characterize evaluation methods for inference-time RAG and GraphRAG systems in healthcare.
• * To analyze how evaluation constructs are defined, operationalized, and reported across different system layers and settings.

Main Methods:

• * Conducted a scoping review following PRISMA-ScR guidelines, searching multiple databases up to May 2026.
• * Included studies on healthcare-relevant LLM systems using inference-time RAG with at least one evaluation component.
• * Extracted data on study characteristics, system design, retrieval, evidence linkage, safety, GraphRAG specifics, reporting, and governance.

Main Results:

• * 157 studies were included, primarily focusing on clinical question answering and decision support.
• * Most evaluations were offline (89.2%), with limited workflow-facing or prospective assessments.
• * Significant gaps were identified in evaluating retrieval quality, fine-grained evidence verification, safety, LLM-as-judge bias, and GraphRAG components in real-world settings.

Conclusions:

• * Healthcare RAG and GraphRAG system evaluation is rapidly growing but lacks consistent reporting and definitions.
• * A shift towards more transparent, layer-specific, safety-oriented, and clinically contextualized evaluations is needed.
• * Addressing identified gaps is critical for advancing the clinical implementation of these AI systems.