Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in...

Area of Science:

• Artificial Intelligence
• Biomedical Informatics
• Natural Language Processing

Background:

• Assessing the run-to-run reproducibility of advanced AI models is crucial for reliable application in scientific research.
• Gemini 3 Flash Preview and GPT-5.2 were evaluated for their consistency in classifying trial success.
• The study investigated reproducibility across various operational parameters, including temperature and reasoning settings.

Purpose of the Study:

• To quantify the run-to-run reproducibility of Gemini 3 Flash Preview and GPT-5.2 for binary trial-success classification.
• To determine if single-run outputs are sufficient for accurate classification across different model configurations.
• To assess the impact of temperature and reasoning/thinking settings on model consistency.

Main Methods:

• Utilized 250 biomedical trial abstracts, pre-labeled for primary endpoint success.
• Evaluated Gemini across four thinking levels (minimal to high) and temperatures (0.0-2.0).
• Assessed GPT-5.2 across multiple reasoning-effort levels and conducted a temperature sweep with reasoning disabled; each setting was run three times.

Main Results:

• High run-to-run reproducibility was observed for both Gemini (κ = 0.942-1.000) and GPT-5.2 (κ = 0.984-0.995).
• Invalid output rates were minimal (0%-1.5% for Gemini, 0% for GPT-5.2).
• Classification performance (F1 score) remained stable (0.955-0.971), with minor improvements from majority voting.

Conclusions:

• Gemini and GPT-5.2 demonstrate high reproducibility in binary biomedical classification tasks with constrained outputs.
• Single-run model executions are frequently adequate for reliable classification.
• Minimal replication can serve as a practical method for verifying stability and ensuring result integrity.