Program provider and observer ratings of school-based preventive intervention implementation: agreement and relation...

Few prevention studies have examined the degree to which different measures of program implementation adherence predict youth outcomes. The current study was conducted with rural middle school youth participating in a longitudinal school-based preventive intervention program. Study participants' average age at the pretest assessment was 12.3 years. The association between program implementation ratings supplied by provider self-reports and trained independent observer reports were evaluated. In addition, the relationship between measures of implementation and youth outcomes were examined. Results indicated that although program providers tended to report higher implementation than independent observers, most ratings were correlated significantly across raters. Observer-reported implementation ratings significantly predicted several youth substance-related outcomes, while provider-reported self-ratings did not. Program provider characteristics predicted several youth outcomes. Findings suggest that there might be a social desirability bias in provider self-reported ratings of implementation and that caution must be used when interpreting self-reported ratings of implementation.