A causal framework for explaining effect heterogeneity in conceptual replications

Area of Science:

• Social Psychology
• Methodology

Background:

• Intervention effects often vary across replication studies, but the reasons for this heterogeneity, particularly due to differing study implementations, are not well understood.
• Identifying how specific study characteristics influence results is crucial for inferring research practice impact and building theory.

Purpose of the Study:

• To present a statistical approach for identifying the causal effects of study characteristics on treatment effects in randomized experiments.
• To address situations where unintended differences in study implementation across replications cannot be avoided.

Main Methods:

• Formal definitions of causal effects and identification assumptions are provided.
• The approach derives causal estimands for study-specific treatment effects.
• Methods for testing assumptions and illustrating consequences of unmet assumptions are included.

Main Results:

• The proposed statistical approach enables the identification of how study characteristics influence treatment effects, even with implementation variations.
• Assumptions are more likely to be met in controlled settings like prospective replication or many-lab studies.
• An empirical example using the imagined intergroup contact effect illustrates the application of the method.

Conclusions:

• The developed statistical method offers a way to disentangle the impact of study characteristics on intervention effects.
• This approach enhances the interpretation of replication studies and contributes to robust theory development in social psychology.
• It provides a framework for more rigorous analysis when comparing results across studies with implementation differences.