Issues of rigor and feasibility when observing the quality of program implementation: a case study.

Area of Science:

• Program Evaluation
• Educational Research
• Implementation Science

Background:

• Measuring implementation quality via observations involves balancing rigor (reliability, validity) with feasibility.
• Limited literature addresses this tradeoff in program evaluation practice.
• Case example focuses on a small education professional development program.

Purpose of the Study:

• To present a case example of addressing rigor and feasibility in observation development.
• To demonstrate methods for measuring implementation quality of professional development.
• To analyze the reliability and feasibility of observation tools.

Main Methods:

• Developed and conducted observations for implementation quality measurement.
• Utilized meta-evaluative analyses to assess observation reliability.
• Focused on a small-scale education professional development program.

Main Results:

• Feasibility constraints did not significantly diminish the rigor of the observation methods.
• Meta-evaluative analyses confirmed the reliability of the quality observations.
• Demonstrated a practical approach to rigorous program evaluation.

Conclusions:

• Rigorous and feasible observation methods are achievable in program evaluation.
• Balancing rigor and feasibility is crucial for effective implementation quality assessment.
• Findings offer practical guidance for evaluators facing similar constraints.