Is replication possible without fidelity?

Area of Science:

• Psychology
• Research Methodology
• Scientific Replication

Background:

• Replication failures in experimental psychology are a growing concern.
• Existing explanations include p-hacking, publication bias, and hidden moderators.
• Reduced experimental power due to protocol deviations is an under-explored factor.

Purpose of the Study:

• To systematically investigate how manipulating experimental fidelity impacts effect size.
• To identify statistical patterns in replication attempts that may indicate fidelity issues.

Main Methods:

• A series of computer simulations were conducted.
• The simulations manipulated fidelity to experimental protocols to observe effects on effect size.

Main Results:

• Fidelity violations were found to significantly influence effect size.
• The observed statistical patterns mimicked those seen in large-scale replication projects (e.g., ManyLabs) and meta-analyses.
• This suggests that deviations from experimental protocols are common in psychology replication attempts.

Conclusions:

• Violations of experimental fidelity are a plausible cause of replication failures in psychology.
• Adopting methods for improving and measuring fidelity, as used in intervention science, medicine, and education, is recommended for psychology.
• Enhanced fidelity management can improve the reliability of psychological research.