p-Curve and p-Hacking in Observational Research

Area of Science:

• Statistical methodology
• Econometrics
• Epidemiology

Background:

• The p-curve, a distribution of significant p-values, is used to estimate true effect sizes and detect p-hacking in published literature.
• Existing methods assume minimal bias, potentially limiting their applicability in real-world research scenarios.

Purpose of the Study:

• To analyze the reliability of p-curves in observational research when p-hacking is present.
• To investigate the impact of omitted-variable bias on p-curve analyses.

Main Methods:

• Simulations were conducted to model p-curves under varying degrees of p-hacking and omitted-variable bias.
• A practical example evaluated the relationship between malaria prevalence and economic growth (1960-1996) using p-curve analysis.

Main Results:

• Simulations demonstrated that p-curves from true effects and p-hacked null effects are difficult to distinguish, even with minor bias.
• The malaria prevalence and economic growth example illustrated the practical challenges in interpreting p-curves due to potential confounding.

Conclusions:

• The p-curve method's validity is questionable in observational research with potential p-hacking and omitted-variable bias.
• Empirical calibration of p-values may be necessary for reliable interpretation in observational studies.
• Similar unreliability concerns may extend to experimental studies with randomization violations.