A Design-Based Approach to Improve External Validity in Welfare Policy Evaluations.

Area of Science:

• Social Sciences
• Policy Evaluation
• Experimental Design

Background:

• Large-scale randomized experiments are crucial for assessing policy intervention impacts on average outcomes.
• Improving the external validity of such experiments is an active area of research.
• Balanced sampling offers a novel approach to site selection, bypassing random sampling and accommodating practical recruitment issues like high nonresponse.

Purpose of the Study:

• To explore the implementation of a balanced sampling strategic site selection method.
• To evaluate its applicability within a welfare policy evaluation context.

Main Methods:

• Balanced sampling aims to create a sample compositionally similar to the target inference population.
• This involves developing a population frame, stratifying it, and ranking units within strata.
• Ranking identifies potential replacement sites to mitigate nonresponse challenges.

Main Results:

• Developing a suitable population frame presented challenges, with three viable options identified for welfare policy.
• A recruitment plan was formulated incorporating study-specific contextual variables to manage nonresponse.

Conclusions:

• The balanced sampling method offers a strategic approach to site selection in policy evaluations.
• It provides a framework for addressing practical recruitment challenges, particularly nonresponse, thereby enhancing external validity.