Campbell's Law Explains the Replication Crisis: Pre-Registration Badges Are History Repeating

Area of Science:

• Psychological science
• Research methodology
• Scientific integrity

Background:

• The replication crisis in science highlights issues with research practices.
• Campbell's Law describes how performance indicators can be distorted when pursued as goals.
• Existing tools like p-values and hypotheses have been misused, contributing to fragile findings.

Purpose of the Study:

• To examine whether pre-registration mandates, such as pre-registration badges (PRBs), are an effective solution to the replication crisis.
• To argue that PRBs, like other tools, may be distorted and harm scientific progress.
• To propose alternative solutions for fostering robust science.

Main Methods:

• Conceptual analysis applying Campbell's Law to pre-registration mandates.
• Review of existing evidence on the use and misuse of pre-registration badges.
• Argumentation based on theoretical principles and observed trends in scientific practice.

Main Results:

• Pre-registration mandates, like PRBs, risk becoming goals in themselves, leading to distorted use.
• Evidence suggests that papers seeking PRBs may violate their spirit and rules.
• Pre-registration mandates may discourage optimal practice, worsen the file drawer problem, and encourage 'pre-registering after results are known' (PRARKing).

Conclusions:

• Pre-registration mandates may create a false sense of trust in unreliable findings.
• A combination of design features (sample size, measurement, robustness checks, pre-registration) supports replicability, but none should be canonized.
• The most crucial solution is a sociocultural shift towards valuing and reinforcing robust science over "flashy but fragile" science.