The reliability of replications: a study in computational reproductions

Area of Science:

• Computational social science
• Research methodology

Background:

• Reproducibility is crucial for scientific integrity.
• Computational reproduction, or replicating analyses with code, faces challenges.
• Variability in researcher approaches can impact replication success.

Purpose of the Study:

• To investigate researcher variability in computational reproduction.
• To compare reproduction success rates between transparent and opaque conditions.
• To identify causes of error in computational replication workflows.

Main Methods:

• 85 independent teams attempted to numerically replicate an original study on policy preferences and immigration.
• Teams were divided into 'transparent' (received code and data) and 'opaque' (received only methods and results) groups.
• Qualitative analysis of workflows was conducted to identify error sources.

Main Results:

• The transparent group achieved higher success rates in verifying original results (95.7%) compared to the opaque group (89.3%).
• Exact numerical reproductions to the second decimal place were less common, especially in the opaque group (48.1%).
• Mistakes and procedural variations were identified as key causes of error, with transparency aiding reliable success.

Conclusions:

• Transparency, including access to original code and data, is essential for reliable computational reproduction.
• Improved training, institutional checks, and greater awareness of research complexity are needed.
• Reducing subjective difficulty in reproduction tasks can enhance researcher engagement and success.