Accumulating evidence across studies: Consistent methods protect against false findings produced by p-hacking

Area of Science:

• Methodology
• Statistical Inference
• Empirical Research

Background:

• Empirical science frequently evaluates competing explanations for observed data.
• "Significant" p-values in statistical testing typically indicate the implausibility of a null (zero) effect.
• Concerns exist regarding p-hacking, or flexible analysis strategies, potentially inflating false positive rates.

Purpose of the Study:

• To investigate the impact of p-hacking on the reliability of findings across multiple studies.
• To assess whether consistent methodologies across studies mitigate the effects of p-hacking.
• To evaluate the plausibility of p-hacking as an explanation for consistent empirical results.

Main Methods:

• Conducted simulations of study sets to model the effects of p-hacking.
• Compared false finding rates for single studies versus sets of studies with consistent methods.
• Examined the influence of selective reporting and varying degrees of p-hacking.

Main Results:

• Consistent methods across studies dramatically reduce the potential for p-hacking to yield significant results.
• P-hacking requires substantial selective reporting and severe, intentional manipulation to produce consistent false findings across studies.
• P-hacking can generate high false positive rates in single, large-sample studies but is less effective in methodologically consistent study series.

Conclusions:

• Methodologically consistent study sets are robust against p-hacking, enhancing the reliability of empirical findings.
• P-hacking is an unlikely explanation for consistent results across multiple studies with uniform methods.
• Series of studies with consistent methods offer greater protection against false positives than single, large-sample studies.