Some properties of p-curves, with an application to gradual publication bias

p-curves provide a useful window for peeking into the file drawer in a way that might reveal p-hacking (Simonsohn, Nelson, & Simmons, 2014a). The properties of p-curves are commonly investigated by computer simulations. On the basis of these simulations, it has been proposed that the skewness of this curve can be used as a diagnostic tool to decide whether the significant p values within a certain domain of research suggest the presence of p-hacking or actually demonstrate that there is a true effect. Here we introduce a rigorous mathematical approach that allows the properties of p-curves to be examined without simulations. This approach allows the computation of a p-curve for any statistic whose sampling distribution is known and thereby allows a thorough evaluation of its properties. For example, it shows under which conditions p-curves would exhibit the shape of a monotone decreasing function. In addition, we used weighted distribution functions to analyze how 2 different types of publication bias (i.e., cliff effects and gradual publication bias) influence the shapes of p-curves. The results of 2 survey experiments with more than 1,000 participants support the existence of a cliff effect at p = .05 and also suggest that researchers tend to be more likely to recommend submission of an article as the level of statistical significance increases beyond this p level. This gradual bias produces right-skewed p-curves mimicking the existence of real effects even when no such effects are actually present. (PsycINFO Database Record