Risk of Bias in Experiments, Quasi-Experiments and Natural Experiments Across Disciplines: Discussion Paper and...

The evidence we provide to support decision-making should be rigorously appraised so that the findings are shown to be valuable. We discuss the risk of bias in impact evaluations on social and natural science topics - that is, studies using a variety of experimental, quasi-experimental and natural experimental approaches to quantify the causal effect of an intervention, program or policy on an outcome of interest. Existing tools to facilitate evaluation of the risk of bias are usually conceptualized to assess either randomized controlled trials (RCTs) or non-randomized studies of interventions, often called quasi-experimental designs (QEDs) or natural experimental evaluations, but not both. The tools do not adequately reflect how common sources of bias might be addressed in impact evaluations in the social and natural sciences, or the bias sources particular to certain types of design, such as participant reactivity to researcher observation in a trial, or when modelling incorporates known selection mechanisms other than randomization, such as subversion of the assignment rule in a discontinuity design. We present a heuristic to assist reviewers in assessing the confidence in causal inferences. Our approach emphasizes four common sources of bias across RCTs, QEDs and natural experiments - the equivalence of groups, the fidelity of study conditions, the adequacy of measurement, and the reporting of analyses - and we provide signaling questions to evaluate these sources of bias for particular types of study. The approach should be adapted to suit interventions and review topics of interest.