Clinical trials are often false positive: a review of simple methods to control this problem

Statistical hypothesis testing is much like gambling. If, with one statistical test, your chance of a significant result is 5%, then, after 20 tests, it will increase to 40%. This result is based on the play of chance. In current clinical trials, instead of a single efficacy-variable of one treatment, multiple efficacy-variables of more than one treatment are increasingly assessed.

The current paper reviews some methods for reducing the problem of falsely positive results due to multiple testing.

These methods include (1) the Bonferroni test, (2) the least significant difference (LSD) test, (3) other less conservative, more rarely used methods like Tukey's honestly significant difference (HSD) test, Dunnett's test, Student-Neuman-Keuls test, Hochberg's adjustment, and the Hotelling Q-square test. Alternative approaches to the problem of multiple testing include (4) the construct of composite endpoints, (5) no adjustment at all, but a more philosophical approach to the interpretation of the p-values, and (6) the replacement of the traditional 5% rejection level with a 1% rejection level or less. Evidence-based medicine is under pressure, because trials do not adequately apply to their target populations. As long as the effects of multiple testing are not routinely assessed in the analysis of clinical trials, it can not be excluded as one of the mechanisms responsible.