Flaws in evaluations of social programs: illustrations from randomized controlled trials.

Area of Science:

• Social Science Research Methods
• Evaluation Science

Background:

• Identifies eight common flaws impacting the validity of impact evaluations.
• Discusses how these flaws affect impact estimates, categorized into statistical imprecision, biases, treatment fidelity, and design weakening.

Purpose of the Study:

• To describe and categorize common flaws in impact evaluations.
• To illustrate these flaws with examples from social experiments, including randomized controlled trials (RCTs).
• To provide guidance on minimizing these flaws.

Main Methods:

• Categorization of eight flaws based on their effect on impact estimates.
• Illustration of flaws using examples from social experiments and randomized controlled trials.
• Presentation of a summary table detailing flaws, circumstances, seriousness, and minimization strategies.

Main Results:

• Identifies frequently occurring flaws: response bias from attrition, inadequate treatment implementation, and insufficient sample size.
• Highlights that some flaws lead to minor issues, while others can invalidate the entire evaluation.
• Explains contributing factors to small sample sizes, such as marketing, target group size, and voluntary participation.

Conclusions:

• Many identified flaws in impact evaluations can be minimized through proactive measures.
• Implementation failures and small sample sizes are often preventable with adequate planning.
• Response bias can frequently be mitigated, for instance, by enhancing survey follow-up efforts.