Using a Satisficing Model of Experimenter Decision-Making to Guide Finite-Sample Inference for Compromised...

Area of Science:

• Econometrics
• Social Experimentation
• Statistical Inference

Background:

• Social experiments often face challenges with imperfect randomization protocols.
• Administrative constraints can lead experimenters to "satisfice" in achieving covariate balance.
• Partially documented or re-randomization designs complicate accurate analysis.

Purpose of the Study:

• To present a decision-theoretic economic framework for analyzing social experiments with compromised random assignment.
• To develop robust hypothesis testing methods for small samples under imperfect randomization.
• To re-examine the effectiveness of interventions like the Perry Preschool Project using a novel analytical approach.

Main Methods:

• A decision-theoretic economic model is employed to analyze flawed experimental designs.
• Design-based small-sample hypothesis tests are developed using worst-case randomization null distributions.
• The methodology is applied to re-evaluate estimates from the Perry Preschool Project.

Main Results:

• The proposed approach accommodates various compromised experimental designs, including imperfectly documented re-randomization.
• Worst-case null distributions provide a conservative basis for hypothesis testing in flawed experiments.
• Re-analysis of the Perry Preschool Project highlights the critical impact of modeling reassignment on effectiveness estimates.

Conclusions:

• A flexible and robust economic framework can effectively analyze social experiments with compromised randomization.
• The choice of modeling reassignment significantly influences the conclusions drawn about program effectiveness.
• This approach offers valuable tools for researchers dealing with imperfectly executed social experiments.