Generalizability of heterogeneous treatment effect estimates across samples

Area of Science:

• Social Sciences
• Political Science
• Survey Methodology

Background:

• Generalizability of survey experiments to target populations is crucial.
• Nonrepresentative convenience samples are widely used in research.
• Treatment effect heterogeneity is a key factor influencing generalizability.

Purpose of the Study:

• To investigate the extent to which subgroup treatment effect estimates generalize from original studies to replication studies using convenience samples.
• To assess the role of treatment effect heterogeneity in explaining the correspondence between nationally representative experiments and convenience sample studies.

Main Methods:

• Analysis of subgroup conditional average treatment effects.
• Utilized 27 original-replication study pairs with 101,745 individual survey responses.
• Compared effect estimates between nationally representative and convenience samples.

Main Results:

• Overwhelming pattern of treatment effect homogeneity observed across studies.
• Subgroup effect estimates showed considerable generalization.
• Treatment effect homogeneity partially explains the strong correspondence in average treatment effects.

Conclusions:

• Treatment effect homogeneity is a primary reason for the generalizability of survey experiments using convenience samples.
• Findings support the use of convenience samples when treatment effects are homogeneous.
• Further research can explore specific conditions that might lead to heterogeneity.