Balanced examination of positive publication bias impact

Positive publication bias is the tendency to favor studies that reject null hypotheses for publication and is widely regarded as detrimental to research enterprise quality. However, this view oversimplifies the interplay between selection biases and the statistical properties of hypothesis testing.

Using theoretical models under varying assumptions about the research landscape in terms of the truth values of hypotheses investigated, we examine how increasing bias towards publishing studies with statistically significant results over those without inflates false positive rate while improving true positive rate of studies found in published literature.

We demonstrate that even when most null hypotheses being investigated in a research field are true, a preference for publishing studies with statistically significant results when choosing among many articles that are each investigating different topics substantially increases the proportion of published true positives while only modestly inflating the proportion of published false positives. This is true so long as individual research studies are being conducted at reasonable significance level and power relative to the research landscape.

Observed positive publication bias based on the proportion of published studies with positive results compared to unpublished studies is not inherently harmful to the research enterprise when publishers and researchers are acting competently and in good faith.