Statistical Uncertainty Analysis for Small-Sample, High Log-Variance Data: Cautions for Bootstrapping and Bayesian...

Area of Science:

• Computational Chemistry and Physics
• Statistical Mechanics
• Biophysics

Background:

• Molecular simulations enable calculation of complex observables like protein folding rate constants.
• These simulations are computationally intensive, often yielding small datasets with high variance.
• Standard statistical methods and bootstrapping can produce unreliable confidence intervals for such data.

Purpose of the Study:

• To evaluate the suitability of standard and Bayesian bootstrapping for analyzing small-sample, high log-variance data from molecular simulations.
• To identify biases in bootstrapping methods when applied to challenging datasets.
• To recommend appropriate statistical approaches for uncertainty quantification in computational studies.

Main Methods:

• Analysis of model distributions and reanalysis of atomistic simulation data.
• Comparison of standard bootstrapping and Bayesian bootstrapping strategies.
• Evaluation of confidence interval reliability and systematic biases.

Main Results:

• Standard bootstrapping yields unphysical confidence intervals and exhibits systematic bias in log space.
• The Bayesian bootstrap provides more reliable uncertainty intervals but still suffers from limitations with small, high-variance samples.
• Neither method can fully overcome the challenge of underestimating means from such datasets.

Conclusions:

• The Bayesian bootstrap is a more consistent alternative to standard bootstrapping for molecular simulation data.
• Caution is advised when interpreting uncertainty intervals from both methods, especially with small, high-variance samples.
• Findings are applicable to rate constants and potentially other nonlinear averages in computational studies.