The false evidence rate: An approach to frequentist error rate control conditioning on the observed P value

A P value is conventionally interpreted either as a) the probability by chance of obtaining more extreme results than those observed or b) a tool for declaring significance at a prespecified level. Both approaches carry difficulties: b) does not allow users to make inferences based on the data in hand, and is not rigorously followed by researchers in practice, while (a) is not meaningful as an error rate. Although P values retain an important role, these shortcomings are likely to have contributed significantly to the scientific reproducibility crisis. We introduce the concept of defining long-run frequentist error rates given the observed data, allowing researchers to make accurate and intuitive inferences about the probability of making an error after proposing that the null hypothesis is false. As one approach, we define the false evidence rate (FER) as the probability, under the null hypothesis, of observing a hypothetical future P value providing evidence toward the alternative hypothesis suggested by the observed P value, which we define as a false positive. FERs are much more conservative than their corresponding P values, consistent with studies demonstrating that the latter do not effectively control error rates across the scientific literature. To obtain an FER below 5%, one needs to obtain a P value below approximately [Formula: see text], while a P value of 5% corresponds to an FER of about 25%.