A longitudinal perspective on monitoring outcomes of an innovative program

The use of outcome assessment to evaluate the performance of programs over many years of operation is becoming an increasingly important aspect of health care management. Over a five-year period of program monitoring, this study examined changes in individual client outcomes three months after discharge from a residential work therapy program for veterans with severe substance use disorders. The study also examined the relationship between these outcomes and changing program features, such as staffing, treatment variables, and follow-up rates.

Data on admissions characteristics, services delivered during treatment, and status at discharges were collected for 3,390 veterans who were treated in 25 sites in the Department of Veterans Affairs' Compensated Work Therapy/Transitional Residence program. Follow-up data were gathered three months after discharge for 1,771 veterans (52.2 percent). Hierarchical linear modeling was used to examine the association between year of discharge, site-level measures of program staffing and follow-up rate, and individual patient-level treatment variables and outcomes.

Over the five-year monitoring period, site staff-to-bed ratios and follow-up rates dropped substantially, and veterans attended more Alcoholics Anonymous and Narcotics Anonymous meetings and had more toxicology screens. Higher staff-to-bed ratios were associated with more positive employment outcomes, and higher follow-up rates were associated with poorer outcomes in substance abuse and lower total income. However, no significant outcome trends were observed in clinical follow-up measures after the analyses adjusted for these factors, suggesting that program effectiveness did not deteriorate during a period of program change.

Long-term evaluations of program process and follow-up status can usefully document sustained program effectiveness over many years. Such efforts should enhance their value by examining and adjusting for the impact of factors such as changing staffing levels and follow-up rates.