A Tutorial on Hunting Statistical Significance by Chasing N.

Area of Science:

• Psychology
• Cognitive Neuroscience
• Research Methodology

Background:

• Replicability crisis in psychological and cognitive neuroscience studies.
• Data dredging (p-hacking) significantly increases Type I errors, leading to false positives.
• Existing 5% false positive rate is substantially exceeded by p-hacking.

Purpose of the Study:

• To systematically illustrate the impact of data dredging techniques on false positive findings.
• To develop intuition for avoiding, detecting, and criticizing common data dredging problems.
• To demonstrate how easily bias can be introduced into research data.

Main Methods:

• Simulation of data dredging techniques using true null distributions.
• Illustration of violating stopping rules by repeatedly checking statistical significance.
• Demonstration of post-hoc participant grouping along unplanned variables.
• Analysis of bias introduced by mild data selection and re-testing.

Main Results:

• Data dredging techniques substantially boost false positive rates.
• Violating stopping rules by altering participant numbers inflates false positives.
• Post-hoc grouping of participants increases false positives.
• Mild data selection and re-testing easily introduce strong bias.

Conclusions:

• Common data dredging practices can lead to 20-50% or more false positives.
• Researchers must be vigilant against p-hacking to maintain scientific integrity.
• Understanding these techniques is crucial for robust and reproducible research.