External Validity in Policy Evaluations that Choose Sites Purposively.

Area of Science:

• Program Evaluation
• Social Science Research Methods

Background:

• Social program impact evaluations frequently utilize multiple sites for data collection.
• Site selection is often purposive and nonrandom, potentially compromising representativeness.

Purpose of the Study:

• To propose a conceptual model for understanding purposive site selection bias in program evaluations.
• To derive a mathematical expression for bias in pooled impact estimates resulting from purposive sampling.

Main Methods:

• Conceptual modeling of purposive site selection.
• Mathematical derivation of bias in pooled impact estimates.

Main Results:

• Purposive site selection can be viewed as a random sample from an unknown population with varying probabilities.
• A formal expression quantifies the bias in impact estimates due to nonrandom site selection.

Conclusions:

• Understanding the bias from purposive site selection is crucial for accurate program evaluation.
• Further research is needed to quantify bias in practice and improve the external validity of evaluations.