Testing for Response Shift Bias in Evaluations of School Antibullying Programs

Involvement in bullying at school is detrimental to students' mental and physical health; however, school antibullying programs have not been found to be uniformly successful. Self-reported frequency of involvement in bullying victimization and perpetration, often used as outcome measures in intervention impact evaluation studies, may be subject to response shift, particularly in intervention conditions. Such differential shifts could lead to biased estimates of intervention effects.

This study investigated the presence of reconceptualization, reprioritization, and recalibration response shift, resulting from intervention implementation.

The study subjects were Grade 8 students (n = 3,382, 53% female) in the 35 schools participating in the Cyber Friendly Schools Project, a longitudinal group-randomized intervention trial. Response shift was assessed by comparing traditional and retrospective pretest measures of bullying involvement, as well as testing for measurement invariance over time in the Forms of Bullying Scale (FBS) using confirmatory factor analyses.

No evidence of response shift was found, indicating students' understandings of bullying behavior remained stable over time. These findings also demonstrate the applicability of the FBS in longitudinal studies involving adolescents.

While response shift was not present in our study, researchers conducting program evaluations in other contexts are advised to consider testing for this potential source of bias in their studies.