Efficient algorithm for "on-the-fly" error analysis of local or distributed serially correlated data

Area of Science:

• Computational Physics
• Statistical Mechanics
• Numerical Analysis

Background:

• Accurate estimation of statistical errors is crucial for Monte Carlo simulations.
• Existing methods like the Flyvbjerg-Petersen blocking method have limitations in efficiency and dynamic application.
• Determining convergence criteria before lengthy computations is often challenging.

Purpose of the Study:

• To introduce the Dynamic Distributable Decorrelation Algorithm (DDDA) for efficient, on-the-fly statistical error calculation.
• To improve upon existing methods for handling serially correlated data in computational studies.
• To enable dynamic termination of Monte Carlo simulations based on achieved convergence.

Main Methods:

• Development of the Dynamic Distributable Decorrelation Algorithm (DDDA).
• Implementation of DDDA for on-the-fly calculation of expectation value errors.
• Design of a parallel implementation of DDDA with low communication overhead.
• Application of DDDA to Quantum Monte Carlo calculations for variance evaluation.

Main Results:

• DDDA efficiently calculates true statistical errors of expectation values from serially correlated data.
• The algorithm allows for dynamic, on-the-fly termination of Monte Carlo calculations upon reaching desired convergence.
• DDDA supports parallel implementations with minimal communication complexity (O(log(2)N)).
• On-the-fly variance calculation is demonstrated for both serial and parallel Quantum Monte Carlo simulations.

Conclusions:

• DDDA offers a significant improvement over traditional methods for statistical error estimation in serially correlated data.
• The dynamic termination capability of DDDA can drastically reduce computational costs for achieving precise results.
• DDDA's parallel efficiency and on-the-fly variance calculation make it a valuable tool for large-scale simulations.