Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects.

Area of Science:

• Psychology
• Economics
• Reproducibility Research

Background:

• Large-scale replication projects (RPs) indicate concerningly low reproducibility rates.
• Substantial shrinkage of effect size is observed, with replication effect sizes being smaller than original ones.
• Heterogeneity in study design, measures, and populations across study pairs contributes to variation beyond statistical noise.

Purpose of the Study:

• To rigorously analyze the amount and sources of shrinkage and heterogeneity in RPs.
• To investigate factors driving high or low reproducibility rates.
• To contextualize the results of RPs by understanding variability.

Main Methods:

• Utilizing meta-analysis methodology to quantify heterogeneity in RPs.
• Applying meta-regression to investigate sources of shrinkage and heterogeneity.
• Proposing location-scale meta-regressions to directly link study characteristics with shrinkage (location) and heterogeneity (scale).

Main Results:

• The proposed location-scale meta-regression approach effectively relates study characteristics to shrinkage and heterogeneity.
• This methodology provides insights into the drivers of varying reproducibility rates.
• Analysis applied to data from Replication Project Psychology and Replication Project Experimental Economics.

Conclusions:

• Location-scale meta-regressions offer a powerful tool for understanding reproducibility.
• Identifying factors influencing shrinkage and heterogeneity is crucial for interpreting RP findings.
• The proposed methods enhance the contextualization and understanding of scientific reproducibility.