Don't Let Your Analysis Go to Seed: On the Impact of Random Seed on Machine Learning-based Causal Inference

Area of Science:

• Epidemiology
• Biostatistics
• Computer Science

Background:

• Machine learning (ML) enhances causal effect estimation in epidemiology, reducing reliance on strict model assumptions.
• The inherent randomness in ML algorithms can lead to results influenced by the initial random seed.
• Doubly robust estimators are a popular ML approach for causal inference.

Purpose of the Study:

• To investigate the influence of random seeds on doubly robust estimators in causal effect estimation.
• To develop and validate methods for stabilizing ML-based causal estimates against random seed variability.
• To provide practical guidance for minimizing seed-related bias in epidemiological research.

Main Methods:

• Examined the impact of varying random seeds on doubly robust estimators using a single dataset.
• Developed and implemented techniques to stabilize estimation results across different random seeds.
• Conducted an extensive simulation study to evaluate the effectiveness of stabilization methods.

Main Results:

• Demonstrated that different random seeds can lead to divergent scientific interpretations of causal estimates.
• Showcased that proposed stabilization techniques effectively mitigate seed-related variability.
• Confirmed that stabilization does not compromise the statistical efficiency of the estimators.

Conclusions:

• Random seeds introduce substantial variability in ML-based causal effect estimation, affecting scientific conclusions.
• Stabilization techniques offer a robust solution to ensure reproducible and reliable results in epidemiological studies.
• Researchers should actively address and report random seed influence in ML applications for causal inference.