Comparative evaluation of interrupted time series analytical methods for healthcare quality improvement research: a...

Interrupted Time Series (ITS) analysis is increasingly used to evaluate healthcare quality improvement (QI) interventions, yet limited guidance exists on which analytical approach performs best under the complex data conditions typical of QI research.

A Monte Carlo simulation study was conducted using a full factorial design with 512 unique scenarios (varying time series length, autocorrelation, trend type, seasonality, noise level, intervention effects, external shocks, and covariates), each replicated 500 times, yielding 256,000 simulated datasets. Four models were compared: ARIMAX, Generalized Additive Models (GAM), Prophet, and Long Short-Term Memory (LSTM) networks. Model performance was evaluated using bias, root mean squared error (RMSE), statistical power, and Type I error rate. For counterfactual-based models (Prophet and LSTM) lacking traditional p-values, an empirical null distribution approach was used to derive comparable significance tests.

ARIMAX demonstrated the highest estimation accuracy (RMSE = 7.85) and statistical power for detecting level changes (72.3%), followed by GAM (RMSE = 8.69; power = 70.2%). However, both parametric models exhibited inflated Type I error rates (ARIMAX: 13.8%; GAM: 9.1% for level change), particularly pronounced for slope change detection (ARIMAX: 37.2%). In contrast, Prophet maintained well-controlled Type I error (4.8%) but with substantially lower power (15.4%) and higher estimation error (RMSE = 20.1). LSTM achieved 25.4% power and an RMSE of 14.8 among the 35.5% of simulated datasets in which the model converged; LSTM failed to converge in 64.5% of simulated datasets, predominantly in short series (T = 36 months). Seasonality was the dominant source of estimation bias for parametric models, while trend type was the primary driver for counterfactual models. Model performance degraded progressively with scenario complexity, with power declining from near 100% (simplest conditions) to approximately 50% (most complex conditions) for ARIMAX and GAM.

No single method dominates across all performance criteria. ARIMAX offers the best accuracy and power but risks false positives, particularly for slope change effects. GAM provides a balanced alternative with better Type I error control and robustness to nonlinear trends. Prophet and LSTM are conservative approaches suitable when avoiding false positive conclusions is paramount. These findings provide empirical guidance for selecting ITS analytical methods appropriate to specific QI research contexts and priorities.