Moving beyond the classic difference-in-differences model: a simulation study comparing statistical methods for...

Area of Science:

• Econometrics and Statistical Modeling
• Public Policy Analysis
• Health Services Research

Background:

• State-level policy evaluations are crucial for evidence-based decision-making.
• Difference-in-differences (DID) is a common study design, but statistical model choices vary widely.
• Guidance is needed on optimal statistical models for DID in policy evaluation.

Purpose of the Study:

• To compare the statistical performance of various two-way fixed effect models used in DID.
• To evaluate autoregressive (AR) and generalized estimating equation (GEE) models for DID.
• To identify optimal models for estimating state-level policy effects on opioid mortality rates.

Main Methods:

• Conducted an extensive simulation study motivated by state-level opioid policy evaluations.
• Compared multiple variations of two-way fixed effect models, AR models, and GEE models.
• Assessed performance using bias metrics, root mean squared error, Type I error rates, and power.

Main Results:

• Most linear models showed minimal bias; non-linear and population-weighted models exhibited significant bias (60-160%).
• Linear AR models minimized root mean squared error for mortality rates.
• Many models had high Type I error rates and low power (<10%), risking spurious policy conclusions; linear AR models performed optimally across all metrics.

Conclusions:

• Commonly used DID models have notable limitations for state policy evaluations.
• Autoregressive (AR) models, though rarely used, are optimal for estimating policy impacts.
• Applied researchers should adopt AR models, moving beyond the classic DID framework.