Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions

Area of Science:

• Implementation Science
• Psychometrics
• Public Health Interventions

Background:

• Scaling evidence-based interventions often leads to reduced implementation fidelity and effectiveness.
• Validated fidelity measures are essential for attributing outcomes to interventions and building practitioner confidence.
• A robust model for validating fidelity observation protocols is needed for real-world scaled-up interventions.

Purpose of the Study:

• To provide a model for validating fidelity observation protocols.
• To build evidence of validity for the Session Review Form, an observational tool for interactive drug prevention programs.
• To guide future research on scaled-up evidence-based interventions.

Main Methods:

• Utilized Kane's (2006) assumptions framework for validity evidence (scoring, generalizability, extrapolation, decision).
• Applied confirmatory factor analysis to the Session Review Form data.
• Conducted measurement invariance tests across grade levels and student populations.

Main Results:

• Confirmatory factor analysis supported a two-factor structure: quality of delivery and participant responsiveness.
• Measurement invariance tests indicated the structure was consistent across different student populations and grade levels.
• Findings provide preliminary support for the extrapolation assumption in validity.

Conclusions:

• The validated Session Review Form offers a reliable measure of intervention fidelity for drug prevention programs.
• The proposed validation model can guide future research on fidelity measures for scaled-up interventions.
• Further research is needed to fully establish the validity argument for fidelity measures.