Theory-based evaluation in practice. What do we learn?

Area of Science:

• Program Evaluation
• Social Science Research Methodology

Background:

• Traditional program evaluations often focus on process and outcomes, potentially missing the underlying mechanisms of success or failure.
• Theory-based evaluation (TBE) aims to uncover the 'how' and 'why' behind program results, offering deeper insights.
• The utility and application of TBE in real-world research require examination.

Purpose of the Study:

• To critically examine published studies employing theory-based evaluation (TBE).
• To assess the extent to which TBE provides unique information compared to traditional evaluation methods.
• To identify the strengths, limitations, and lessons learned from applying TBE.

Main Methods:

• Systematic review and analysis of six published theory-based evaluation studies.
• Examination of the explicit and implicit connections between program data and underlying theory.
• Qualitative assessment of the contributions and challenges encountered in each TBE case.

Main Results:

• Authors did not consistently articulate the link between data and program theory.
• TBE successfully identified redundant program elements and intermediate causal pathways.
• Evaluations using TBE raised novel research questions and contributed to methodological shifts.
• Challenges in scaling up pilot programs were illuminated.
• Original program theories were consistently found to be incomplete or inaccurate.

Conclusions:

• Theory-based evaluation (TBE) offers valuable, distinct contributions to understanding program dynamics.
• Clearer articulation of the theory-data link is needed for robust TBE.
• TBE can reveal complexities in program implementation and impact, informing future research and practice.
• The findings suggest iterative refinement of both program theory and evaluation practice is essential.