How best to measure change in evaluations of treatment for substance use disorder

To compare the performance of the Jacobson & Truax (JT) reliable change index (RCI) with three alternative methods, using data from individuals receiving treatment for substance use disorders.

English National Treatment Outcome Monitoring Database for publicly funded specialist community pharmacological and psychosocial interventions.

New adult admissions to treatment across England (1 January-31 December 2008), with in-treatment clinic progress review conducted after an average of 122.8 days for 18,163 individuals.

Self-reported days using heroin, crack, cocaine powder and alcohol during the 4 weeks before admission and clinical review, recorded using the Treatment Outcomes Profile and analysed using a multi-level, mixed-linear model, with both observed and true scores to estimate the effect of regression to the mean (RTM). Differences in performance among the JT RCI and the alternative methods were assessed by the proportion assigned to a reliably 'improved', 'unchanged' or 'reliably deteriorated' category; level of agreement; difference in effect size for observed and true scores; and receiver operating characteristic parameters.

When compared to the alternative methods, the JT RCI was more conservative in assigning individuals to the improved category, and it showed no evidence of inferiority on any measure. For each method, all individuals categorized as reliably deteriorated and the majority of those categorized reliably improved had outcome scores which fell beyond that expected by RTM. Substituting true scores for observed scores moderated the size of the change effect associated with reduced use of the four substances, but this remained statistically significant.

The Jacobson & Truax Reliable Change Index appears to be the optimal measure of change for evaluations of treatment for substance use disorder, in that it is the most conservative for assessing improvement and at least as accurate on all other criteria. Any evaluation of change needs to take account of regression to the mean.