Measuring sustainment of an evidence based treatment for adolescent substance use

Measuring the extent a behavioral treatment is sustained in usual care practice settings after initial support for implementation ends is challenging. Oftentimes dichotomous self-report measures are used to measure health program sustainment, but these measures may not adequately capture the quality or extent of behavioral treatment delivery. Using data collected from community based organizations that received support to implement the Adolescent Community Reinforcement Approach (A-CRA), a measure of the extent of sustainment was derived.

Based on implementation theories and implementation support protocols, a total of 10 core treatment elements were identified to measure the extent of sustainment using information collected from key clinical staff. Item response theory (IRT) and principal component analyses (PCA) were used to further refine the 10 elements into composite measures of sustainment. The association between the 10 elements and the relationships between the comprehensive measures to a dichotomous sustainment measure were also examined.

Results from PCA identified two components from the initial 10 elements for measuring the extent of A-CRA treatment sustainment. The two components described different aspects of organizational support for A-CRA treatment implementation, one representing the quality or extent of treatment staffing and delivery and the other representing the quality of or extent of clinical supervision.

Using IRT and PCA, we were able to derive components that could be used to measure the extent of EBT sustainment and also better capture the quality of treatment delivery than the use of simple dichotomous measure. The methodological contribution of our study is that we have demonstrated a general analytic approach that may be applicable for other psychosocial treatments.