Clinical trials and p-values, beware of the extremes

In randomized controlled trials, prior to statistical analysis, the data are checked for outliers and erroneous data. Statistical tests are, traditionally, not very good at distinguishing between errors and outliers, but they should be able to point out main endpoint results closer to expectation than compatible with random sampling.

To explain from hypothesized and published examples why extreme p-values like p>0.95 and p<0.0001 may indicate that sampling was not completely random.

Extreme p-values can be readily observed in recent issues of high-impact journals. A p-value >0.95 literally means that we have a >95% chance of finding a result less close to expectation and, consequently, a <5% chance of finding a result this close or closer. Often in studies a statistical power of 80% is agreed upon, corresponding with a p-value of approximately 0.01. The ultimate p-value may then be a bit larger or smaller. However, a p-value much smaller than 0.01 will be rarely observed, because it would indicate that the study is overpowered. If the p-values can be assumed to follow a normal distribution around 0.01, then we will have a less than 5% chance of observing a p-value of <0.0001.

In randomized controlled trials, main endpoint p-values larger than p=0.95 will be rare, because they would indicate similarities closer than compatible with a normal distribution of random data samples. Also very low p-values like p<0.0001 will be rarely encountered, because it would mean that the trial was overpowered and should have had a smaller sample size. It would seem appropriate, therefore, to require investigators to explain such results and to consider rejecting the research involved. So far, in randomized controlled trials the null-hypothesis is generally rejected at p<0.05. Perhaps we should consider rejecting the entire study if the main endpoint p-values are >0.95 or <0.0001.