Using machine learning to evaluate treatment effects in multiple-group interrupted time series analysis

Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time, and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. In this paper, we introduce a novel machine learning approach using optimal discriminant analysis (ODA) to evaluate treatment effects in multiple-group ITSA.

We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California (CA) to Montana (MT)-the best matching control state not exposed to any smoking reduction initiatives. We contrast results from ODA to those of ITSA regression (ITSAREG)-a commonly used approach for evaluating treatment effects in ITSA studies.

Both approaches found CA and MT to be comparable on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the post-intervention period (P < 0.0001). The ODA model achieved very high effect strength of sensitivity (a measure of classification accuracy) of 91.67%, which remained high (75.00%) after conducting leave-one-out analysis to assess generalizability.

The ODA framework achieved results comparable to ITSAREG, bolstering confidence in the intervention effect. In addition, ODA confers several advantages over conventional approaches that may make it a better approach to use in multiple group ITSA studies: insensitivity to skewed data, model-free permutation tests to derive P values, identification of the threshold value which best discriminates intervention and control groups, a chance- and maximum-corrected index of classification accuracy, and cross-validation to assess generalizability.