Continuous improvement evaluation: a framework for multisite evaluation studies

This article proposes the use of the continuous improvement evaluation (CIE), a framework for multisite demonstration or evaluation studies. This framework is designed for studying intervention programs that change during the evaluation. The development of family drug courts is provided as an example. CIE relies on outcome data collected over time and benchmarked across similar cases in comparison sites; thus, this study was designed to collect data on effectiveness of intervention programs at multiple sites and over time. A weight is calculated for similarity of any two cases based on features they share. In statistical process control charts, these weights are used to compare outcomes at the site against the average of similar cases in comparison groups. Once data are benchmarked, program staff meet to discuss process changes that have led to improvements in outcomes. To ensure that intervention programs have access to evaluation reports on demand, information technology is used to collect, clean, and pool data. Computers generate study reports, and evaluators review reports after release to clients. Statistical tools can be used to evaluate changing programs. Traditional evaluators may be concerned about some threats to validity associated with CIE. The article concludes with a discussion of typical threats to validity and how these threats are addressed in the CIE framework.