Assessing bias and precision in state policy evaluations: a comparative analysis of time-varying estimators using...

Area of Science:

• Epidemiology
• Health Policy Analysis
• Econometrics

Background:

• Health policy evaluations often assume static treatment effects, but many interventions have dynamic impacts.
• This assumption can lead to inaccurate conclusions about policy effectiveness, especially for time-sensitive issues like opioid overdose.
• Optimal estimation strategies for time-varying policy impacts remain an open methodological question.

Purpose of the Study:

• To evaluate the performance of various panel data estimators in capturing time-varying impacts of state-level opioid policies.
• To compare seven common estimation methods under different time-varying treatment scenarios.
• To provide guidance on selecting appropriate analytical approaches for health policy evaluations.

Main Methods:

• Utilized state-level opioid overdose mortality data (1999-2016).
• Simulated four time-varying treatment scenarios (gradual increase/decline, temporary, inconsistent effects).
• Compared seven estimators: two-way fixed effects event study, debiased autoregressive model, augmented synthetic control, difference-in-differences (DiD) with staggered adoption, event study with heterogeneous treatment, two-stage DiD, and DiD imputation.
• Assessed performance using bias, standard errors, coverage probability, and root mean squared error.

Main Results:

• Estimator performance varied significantly across simulated scenarios.
• Augmented synthetic controls showed lower bias but higher variance when policy effects diminished.
• Difference-in-differences methods offered reasonable coverage in some cases but struggled with non-monotonic effects.
• Autoregressive methods had lower variability but underestimated uncertainty.

Conclusions:

• No single estimator consistently outperformed others across all time-varying policy effect scenarios.
• Researchers must consider the bias-variance tradeoff and expected effect trajectories when selecting methods.
• Careful methodological choices are crucial for accurate attribution of policy effects and valid conclusions in epidemiological policy evaluations, especially for opioid-related interventions.