A statistical framework for evaluating the repeatability and reproducibility of large language models

Systematic evaluation of variability in large language model (LLM)-generated outputs is critical for assessing their reliability. We developed a general statistical framework and Python package (llmReproScore) for quantifying the repeatability and reproducibility of LLM outputs.

Following US Food and Drug Administration terminology, our framework establishes four metrics for quantifying LLM repeatability and reproducibility along two dimensions: semantic and internal. Semantic metrics quantify variability in the meaning of LLM-generated outputs, whereas internal metrics quantify variability in token-generation certainty. We validated these metrics, and as an illustrative use case, applied them to diagnostic reasoning outputs generated across LLMs and prompting strategies using US Medical Licensing Examination questions and 90 real-world rare disease cases from the Undiagnosed Diseases Network.

Validation results showed that semantic scores aligned with human ratings (Spearman ρ = 0.88 for repeatability and 0.66 for reproducibility) and confirmed that internal metrics capture variability in token-generation certainty. In the illustrative use case, LLM repeatability was generally not associated with accuracy, suggesting that LLM output variability and accuracy capture distinct dimensions of model performance.

The proposed metrics can be used to characterize LLM output variability and systematically compare models, prompts, and generation settings. These comparisons can inform more rigorous selection of LLM configurations based on repeatability and reproducibility.

We developed and validated generalizable metrics for quantifying LLM output variability. The proposed metrics can complement accuracy-based evaluation to inform experimental design and support more rigorous comparison and selection of models, prompts, and generation settings.