Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of...

Clinical summary statements convert unstructured patient information into a concise format. They are widely used in virtual patients (VPs) to train medical students in diagnostic reasoning. Most VP systems provide little individualized feedback on the quality of learners' summaries, and earlier automated approaches have required complex natural language processing, limiting their scalability. Large language models (LLMs) may offer a more flexible, rubric-guided way to score summary statements and support feedback. Still, their test-retest reliability and concordance with expert ratings in this context remain unclear.

This study aims to evaluate the test-retest reliability-specifically, whether the models produce stable ratings across repetitions-of LLM-generated medical summary statement ratings, and to measure their concordance with human expert assessments across different prompting strategies.

We conducted an experimental comparative study using 122 rubric-scored summary statements (English and German) from 6 internal medicine VP cases previously rated by 2 expert raters. Statements were evaluated along 4 ternary components (semantic qualifiers, transformation, narrowing, and global rating), and 2 binary components (factual accuracy and person). Four LLMs (Anthropic's Claude, OpenAI's GPT-4o, Mistral AI's Mistral Large, and Mistral AI's Mistral 7B) were each run 5 times under 4 prompting strategies (zero-shot, zero-shot with chain-of-thought [CoT], few-shot, and few-shot+CoT). The test-retest reliability was assessed with intraclass correlation coefficients (ICCs), Fleiss kappa (κ), and concordance with human ratings with accuracy (%) and chance-corrected interrater agreement (Gwet AC1 and AC2).

Large closed-weights models (Claude, GPT-4o, and Mistral Large) showed high test-retest reliability (ICC≥0.92; Fleiss κ=0.59-1.00). Claude was essentially deterministic (ICC and Fleiss κ≥0.99); GPT-4o and Mistral Large reached ICC≥0.97 and Fleiss κ≥0.80 except under zero-shot. Mistral 7B, a small open-weights model, had high ICCs (≥0.88) but low Fleiss κ (0.32-0.52). Few-shot+CoT prompting achieved the best concordance with human ratings for all the models (means across runs and rubric components), peaking for Mistral Large (accuracy 77%; Gwet AC1 and AC2 0.85, 95% CI 0.83-0.87), followed by Claude (71%; 0.77, 0.74-0.81), GPT-4o (67%; 0.72, 0.68-0.77), and Mistral 7B (60%; 0.64, 0.56-0.73). Concordance was higher for the binary rubric components (person and factual accuracy) than for the ternary components. Extreme deviations and invalid outputs were rare but more frequent in ternary components and the smaller local model.

Large, general-purpose LLMs can function as reliable, rubric-guided raters of VP-based clinical summary statements, achieving high concordance with expert ratings under clearly defined criteria in binary rubric components and when using few-shot with CoT prompting. The performance on more nuanced, ternary reasoning dimensions was more variable. LLMs can support scalable, rubric-based formative feedback on clinical reasoning within VP systems but should complement rather than replace human judgment.