Understanding Leisure-related Program Effects by Using Process Data in the HealthWise South Africa Project

As the push for evidence-based programming gathers momentum, many human services programs and interventions are under increased scrutiny to justify their effectiveness across different conditions and populations. Government agencies and the public want to be assured that their resources are being put to good use on programs that are effective and efficient (Guskey, 2000). Thus, programs are increasingly based on theory and evaluated through randomized control trials using longitudinal data. Despite this progress, hypothesized outcomes are often not detected and/or their effect sizes are small (Gingiss, Roberts-Gray, Boerm, 2006). Moreover, findings may go against intuition or "gut feelings" on the part of project staff. Given the need to understand how program implementation issues relate to outcomes, this study focuses on whether process measures that focus on program implementation and fidelity can shed light on associated outcomes. In particular, we linked the process evaluation of the HealthWise motivation lesson with outcomes across four waves of data collection. We hypothesized that HealthWise would increase learners' intrinsic and identified forms of motivation, and decrease amotivation and extrinsic motivation. We did not hypothesize a direction of effects on introjected motivation due to its conceptual ambiguity. Data came from youth in four intervention schools (n = 902, 41.1%) and five control schools (n = 1291, 58.9%) who were participating in a multi-cohort, longitudinal study. The schools were in a township near Cape Town, South Africa. For each cohort, baseline data are collected on learners as they begin Grade 8. We currently have four waves of data collected on the first cohort, which is the focus of this paper. The mean age of the sample at Wave 3 was 15.0 years (SD = .86) and 51% of students were female. Results suggested that there was evidence of an overall program effect of the curriculum on amotivation regardless of fidelity of implementation. Compared to the control schools, all treatment school learners reported lower levels of amotivation in Wave 4 compared to Wave 3, as hypothesized. Using process evaluation data to monitor implementation fidelity, however, we also conclude that the school with better trained teachers who also reported higher levels of program fidelity had better outcomes than the other schools. We discuss the implications of linking process data with outcome data and associated methodological challenges in linking these data.