P-curve: a key to the file-drawer

Area of Science:

• Psychology
• Statistics
• Research Methodology

Background:

• Scientific literature often suffers from publication bias, where only significant results are published.
• P-hacking, or selective analysis, can also create a misleading impression of effect.
• Distinguishing true effects from those arising from selective reporting is a critical challenge.

Purpose of the Study:

• To introduce and validate the p-curve as a novel method for assessing evidential value.
• To provide a tool to differentiate genuine research findings from those influenced by selective reporting.

Main Methods:

• The p-curve is defined as the distribution of statistically significant p values (p < .05) from a body of research.
• Analysis focuses on the skewness of the p-curve: right-skewed curves (more low p values) indicate true effects.
• The method assesses whether selective reporting can be ruled out as the sole explanation for observed results.

Main Results:

• Right-skewed p-curves are diagnostic of evidential value, suggesting genuine effects.
• Non-right-skewed p-curves raise concerns about potential publication bias or p-hacking.
• The p-curve method effectively addresses inferential problems caused by unpublished failed studies.

Conclusions:

• The p-curve offers a robust solution to the problem of selective reporting in scientific research.
• This method enhances the reliability of interpreting research findings by assessing their evidential basis.
• P-curve analysis is crucial for overcoming the limitations of traditional publication and analysis practices.