Differences in observers' and teachers' fidelity assessments

Area of Science:

• Program Implementation Science
• Educational Psychology
• Public Health Interventions

Background:

• Ensuring fidelity in evidence-based programs is critical for successful dissemination and outcomes.
• Discrepancies between self-reported and observer-reported fidelity can impact evaluation accuracy.

Purpose of the Study:

• To compare teacher and observer fidelity ratings in an evidence-based program.
• To assess the relationship between fidelity ratings and program mediators.
• To examine the influence of teaching experience on fidelity.

Main Methods:

• Teachers and trained observers rated program implementation fidelity using parallel forms.
• Fidelity ratings included adherence, quality of delivery, attendance, and engagement.
• Video recordings of 215 classroom sessions were used for observer ratings.

Main Results:

• Teachers reported higher fidelity than observers, indicating a positivity bias.
• Teacher and observer ratings showed low agreement, except for attendance.
• Observer ratings correlated with student pretest scores on program mediators, unlike teacher ratings.

Conclusions:

• Observer fidelity ratings appear more valid and related to program mediators than teacher self-ratings.
• Experienced teachers demonstrated higher fidelity.
• Recommends using both teacher and observer fidelity assessments for comprehensive program evaluation.