Is the Replicability Crisis Overblown? Three Arguments Examined.

Area of Science:

• Psychology
• Statistics
• Scientific Methodology

Background:

• Growing concerns regarding the replicability crisis in scientific research.
• Common arguments downplaying the severity of issues with scientific reproducibility.

Purpose of the Study:

• To critically evaluate three common arguments that suggest concerns about scientific replicability are exaggerated.
• To propose that systematic reforms in scientific practice are necessary to address issues of reproducibility.

Main Methods:

• Analysis of statistical misunderstandings related to alpha levels and their impact on error rates.
• Examination of the interaction between conceptual replication, direct replication, and publication bias.
• Critique of the 'wait and prune' argument for addressing errors in scientific literature.

Main Results:

• The adoption of a low alpha level does not limit the rate of errors entering the scientific literature.
• Conceptual replication, when favored over direct replication, can exacerbate publication bias, leading to a literature with non-existent phenomena.
• The argument that errors will naturally be pruned from the literature over time lacks empirical support.

Conclusions:

• The current level of concern regarding scientific replicability is justified.
• Misinterpretations of statistical principles and biases in publication practices contribute significantly to the problem.
• Systematic reforms in scientific methodology and practice are essential for improving the reliability of research findings.