Using Iterative Experimentation to Accelerate Program Improvement: A Case Example.

Area of Science:

• Program Evaluation
• Implementation Science
• Educational Psychology

Background:

• Experimental evaluation methods can be integrated with design-based implementation research (DBIR), improvement science (IS), and rapid-cycle evaluation (RCE).
• This integration allows for rapid, low-cost, and credible assessments of strategies aimed at improving programs, policies, or practices.

Purpose of the Study:

• To demonstrate the feasibility and benefits of blending DBIR, IS, and RCE practices with embedded randomized controlled trials (RCTs).
• To enhance the pace and efficiency of program improvement initiatives.

Main Methods:

• A two-cycle experimental test of staff-designed strategies for a workforce development program (Year Up's Professional Training Corps - PTC).
• Random assignment of 317 youth across three PTC sites to improvement strategies.
• Primary outcomes measured: completion of the learning and development (L&D) phase and continued college enrollment post-L&D.

Main Results:

• Improvement strategies increased program retention through the L&D phase by nearly 10 percentage points.
• Strategies also increased college persistence following the L&D phase by 13 percentage points.

Conclusions:

• Blending DBIR, IS, and RCE principles with a multi-cycle RCT provides credible efficacy estimates efficiently.
• This integrated approach achieved results within 18 months at modest cost and low burden for program staff.