Statistical Pitfalls in Large Language Model Evaluation: A Practical Guide to Study Design and Analysis

Studies evaluating large language models in medicine increasingly follow a common design in which a fixed set of clinical questions is posed to several sources, such as different language models, clinicians, or patient-education resources. The responses are then scored for outcomes including accuracy, completeness, readability, or similar measures. Because each source answers the same set of questions, the resulting observations are inherently paired: the question functions as the repeated unit across sources. Despite this structure, many studies analyze the resulting data using statistical procedures that assume independent observations.

Treating paired observations as independent introduces a mismatch between the study design and the statistical analysis. In evaluations where multiple sources respond to the same questions, the responses are naturally correlated because they are influenced by shared characteristics of the question, such as difficulty or ambiguity. Ignoring this dependence can reduce statistical power and distort the nominal type I error rate. These issues arise when commonly used independent-sample procedures, including the Mann-Whitney test, the Kruskal-Wallis test, one-way analysis of variance, or unpaired t-tests, are applied to datasets generated from identical question sets. The appropriate analytical framework instead treats the question as the unit of pairing. Methods designed for repeated or paired observations, such as the Friedman test or repeated-measures analysis of variance for overall comparisons, followed by paired contrasts using the Wilcoxon signed-rank test or paired t-tests with appropriate multiplicity correction, provide analyses that align with the underlying design. Effect sizes and regression approaches that account for the paired structure, including mixed-effects models with question treated as a random effect, may further strengthen inference. Recognizing the repeated-measures nature of these evaluations provides a straightforward path toward more statistically coherent analyses.