Evidence that nonsignificant results are sometimes preferred: Reverse P-hacking or selective reporting?

Area of Science:

• Behavioral Ecology
• Scientific Methodology
• Publication Bias

Background:

• Excessive focus on P-values leads to poor scientific practices like selective reporting and P-hacking.
• P-hacking involves manipulating data or analyses to achieve statistically significant results.
• Selective reporting of nonsignificant results and 'reverse P-hacking' are introduced as novel concerning practices.

Purpose of the Study:

• To investigate the occurrence of selective reporting of nonsignificant results and reverse P-hacking in experimental research.
• To determine if these practices lead to an under-representation of significant findings in the literature.
• To test for publication bias towards nonsignificant results in behavioral ecology.

Main Methods:

• Surveyed the behavioral ecology literature, analyzing 250 studies.
• Examined P-values for group differences in potentially confounding variables in randomized controlled experiments.
• Compared observed P-value distributions to expected distributions under the null hypothesis.

Main Results:

• Found a significantly higher proportion of nonsignificant P-values (P > 0.05) than expected by chance (95%).
• The observed under-representation of significant results suggests a potential publication bias.
• This bias may stem from selective reporting of nonsignificant findings or reverse P-hacking.

Conclusions:

• A novel publication bias, favoring nonsignificant results, may exist in the behavioral ecology literature.
• This bias could be driven by researchers selectively reporting nonsignificant findings or engaging in reverse P-hacking.
• Further research is encouraged to test for this bias in other scientific disciplines.