Randomized Controlled Trial Aversion among Public Sector Leadership: A Survey Experiment

Area of Science:

• Social Sciences
• Public Administration
• Program Evaluation

Background:

• Randomized controlled trials (RCTs) are the gold standard for program evaluation.
• Public sector leaders (government and nonprofit decision-makers) infrequently choose RCTs for impact evaluations.

Purpose of the Study:

• To provide descriptive evidence on RCT aversion among public sector leaders.
• To understand factors influencing public sector leaders' likelihood of choosing RCTs for impact evaluations.
• To compare RCT preference patterns between public sector leaders and the general population.

Main Methods:

• A survey experiment was conducted with 2050 public sector leaders and 2060 non-public sector respondents.
• The primary outcome measured was the selection of an RCT as the preferred evaluation option.
• The study examined preferences for RCTs versus quasi-experimental designs.

Main Results:

• The majority of respondents, including public sector leaders, did not select RCTs as their preferred evaluation method.
• Public sector leaders were 13% more likely than the general population to prefer RCTs over quasi-experimental designs.
• Public sector leaders showed decreased likelihood of using RCTs for more intensive interventions.

Conclusions:

• Funders should recognize that public sector leaders often prefer alternatives to RCTs.
• Increased awareness of RCT benefits may enhance their adoption within the public sector.
• Perceived superiority of interventions for control groups may contribute to RCT aversion for intensive programs.