Combining synthetic controls and interrupted time series analysis to improve causal inference in program evaluation

Area of Science:

• Epidemiology
• Biostatistics
• Public Health Policy Evaluation

Background:

• Interrupted time series analysis (ITSA) is a valuable method for evaluating interventions by analyzing outcome trends over time.
• Enhancing ITSA's internal validity often requires a comparable control group, which can be challenging to establish.
• The synthetic controls method (SYNTH) provides a technique for constructing such control groups.

Purpose of the Study:

• To introduce and evaluate a robust framework combining SYNTH and ITSA for intervention effect estimation.
• To assess the effectiveness of California's Proposition 99 in reducing cigarette sales using this combined methodology.

Main Methods:

• The study employed the synthetic controls method (SYNTH) to generate a counterfactual control group for California.
• Interrupted time series analysis (ITSA) regression models were then applied using the synthetic control.
• Covariate balance was assessed to ensure the validity of the synthetic control group.

Main Results:

• Covariate balance was successfully achieved for most covariates between California and its synthetic control.
• A significant decrease in the trend of cigarette sales was observed in California after Proposition 99.
• However, no statistically significant treatment effect on cigarette sales was detected when comparing California to its synthetic control.

Conclusions:

• The combined SYNTH and ITSA framework provides a rigorous method for generating comparable control groups in intervention studies.
• This approach offers enhanced evaluation capabilities, including covariate balance assessment and comprehensive postestimation measures.
• The framework is recommended as a primary approach for evaluating treatment effects in multiple group time series analyses.