Using Decomposed Error for Reproducing Implicit Understanding of Algorithms.

Area of Science:

• Evolutionary Computation
• Machine Learning
• Computational Statistics

Background:

• Reproducibility is crucial for validating evolutionary machine learning (EML) algorithms.
• Current reproducibility checks often rely solely on aggregate prediction error with fixed random seeds, which is insufficient.

Purpose of the Study:

• To introduce an error decomposition framework to enhance the reproducibility of EML results.
• To address limitations of aggregate error scores by ensuring statistical equivalence and confirming algorithmic behavior.

Main Methods:

• Utilized an error decomposition framework estimating bias, internal variance (algorithm-specific), and external variance (data-specific).
• Applied the framework using multiple algorithm runs and diverse training datasets to assess prediction error.
• Analyzed the alignment between expected and actual algorithmic behavior in targeting prediction error reduction.

Main Results:

• The error decomposition framework provides greater certainty in prediction error estimation.
• Decomposition reveals that the actual behavior of some EML algorithms can differ from their expected behavior.
• Identified behavior mismatches in several tested evolutionary algorithms.

Conclusions:

• The proposed framework improves EML reproducibility by ensuring statistical equivalence and enabling behavior confirmation.
• Understanding behavior mismatches is key for refining EML algorithms and their application.
• Error decomposition offers a more comprehensive characterization of EML algorithms beyond aggregate scores.