Are We in Control? How Best to Include a Control Group in Interrupted Time Series Designs: A Simulation Study

While controlled interrupted time series (CITS) are commonly used to evaluate public health policies, how to incorporate control(s) into their statistical modelling has received limited attention. We aimed to compare the statistical performance of different model formulations for including control groups in various segmented regression model specifications (with a particular focus on CITS and Difference-in-Difference [DiD] designs) under conditions where their assumptions are met, as well as when they are violated.

Based on a real-world dataset, we simulated and compared the statistical performance of four model formulations grounded on segmented regressions for including control groups in a pre- and post-evaluation. The compared model formulations were: (1) CITS segmented regression, (2) DiD segmented regression, (3) single ITS of the difference between control and intervention series, and (4) incorporating the control as a covariate in a single ITS. Models were tested across scenarios challenging assumptions around the control group (e.g., non-parallel trends -challenging DiD assumptions-, or inconsistent trend difference over time between groups -challenging CITS assumption-) or regression errors (e.g., heteroscedasticity or autocorrelation). We also included models, including restricted cubic splines of time, which may mitigate distortions from assumption violations. Additionally, we tested for detecting non-parallel trends.

Standard DiD, CITS, and the ITS of the difference between series yielded the lowest bias whenever their design assumptions were satisfied. Overall, including time splines as covariates into ITS of the difference between series achieved the lowest bias and highest coverage also when design assumptions were violated. This makes it a valuable tool for causal inference in settings with parallel, non-parallel or inconsistent trend patterns between groups. Since violations of the trends assumption are often undetectable, methods robust to such violations are extremely valuable.

Modelling CITS as an ITS of the difference between series is among the most robust methods to embed control series into model specifications. Incorporating time splines as model covariates within an ITS of the difference has the potential of reducing bias from assumption violations (including parallel trends) without negative impacts when assumptions hold.