REFINE2: a simplified simulation tool to help epidemiologists evaluate the suitability and sensitivity of effect...

Area of Science:

• Epidemiology
• Biostatistics
• Machine Learning

Background:

• Epidemiologists utilize diverse statistical methods for effect estimation, ranging from regression to advanced machine learning algorithms.
• Choosing the optimal method is challenging due to numerous assumptions, trade-offs, and context-specific performance.
• Evaluating methods via real-data simulations is often impractical for many researchers.

Purpose of the Study:

• To introduce REFINE2, a user-friendly offline Shiny app for comparing statistical estimators in epidemiology.
• To enable analysts to assess algorithm performance within their specific data context for average treatment effect estimation.
• To guide the selection of appropriate statistical models and enhance understanding of finite sample bias.

Main Methods:

• Development of REFINE2, an offline Shiny application for comparative performance evaluation of statistical estimators.
• Automated plasmode simulation to generate a target average treatment effect (ATE) based on observed covariates.
• Assessment of bias and confidence interval coverage for user-specified models against the simulated target ATE.

Main Results:

• The optimal statistical method for effect estimation varied significantly across different data scenarios.
• Suboptimal performance was observed for certain methods under residual confounding.
• REFINE2 demonstrated utility in guiding model selection and understanding limitations like finite sample bias.

Conclusions:

• REFINE2 empowers epidemiologists to select appropriate statistical models by evaluating performance on their own data.
• The app aids in understanding common challenges such as finite sample bias when employing machine learning.
• Context-specific evaluation is crucial for robust effect estimation in epidemiological research.