Human-anchored longitudinal comparison of generative AI with a bias-calibrated LLM-as-judge

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning Evaluation

Background:

• Service large language models (LLMs) evolve rapidly without transparent changelogs, hindering reproducible research and evaluation.
• Assessing LLM performance and stability over time is crucial due to their increasing integration into various applications.

Purpose of the Study:

• To longitudinally track the performance and stability of major service LLM families over time.
• To develop and validate a robust methodology for reproducible LLM evaluation using human and LLM-as-judge assessments.
• To identify patterns of model drift, including stability, improvement, and degradation.

Main Methods:

• A preregistered, human-anchored longitudinal study design with ten weekly evaluation waves.
• Utilized a fixed prompt bank (N=240) across six diverse domains for consistent assessment.
• Employed blinded human raters for correctness judgments and a bias-calibrated LLM-as-judge for pairwise preferences, corrected weekly using a Bradley-Terry model.
• Applied mixed-effects modeling and change-point detection (PELT with MBIC penalty) to identify significant drift patterns.

Main Results:

• Observed divergent stability trajectories across three major LLM families: one consistently stable, one showing improvement, and one exhibiting degradation mid-study.
• Judge calibration enhanced agreement with human judgments (Kendall's τ = 0.59-0.68) and reduced evaluation volatility.
• Safety metrics demonstrated co-variation with detected drift events, suggesting behavioral shifts rather than confirmed causal changes.

Conclusions:

• Service LLMs exhibit distinct and dynamic stability patterns, necessitating continuous and reproducible evaluation frameworks.
• The developed human-anchored, LLM-as-judge methodology offers a scalable and reliable approach for tracking LLM evolution.
• Transparency in LLM development and deployment is essential for fostering trust and enabling accurate performance assessment.