Challenges to multiple testing in clinical trials

Multiple testing problems are complex in evaluating statistical evidence in pivotal clinical trials for regulatory applications. However, a common practice is to employ a general and rather simple multiple comparison procedure to handle the problems. Applying multiple comparison adjustments is to ensure proper control of type I error rates. However, in many practices, the emphasis of the type I error rate control often leads to a choice of a statistically valid multiple test procedure but the common sense is overlooked. The challenges begin with confusions in defining a relevant family of hypotheses for which the type I error rates need to be properly controlled. Multiple testing problems are in a wide variety, ranging from testing multiple doses and endpoints jointly, composite endpoint, non-inferiority and superiority, to studying time of onset of a treatment effect, and searching for minimum effective dose or a patient subgroup in which the treatment effect lies. To select a valid and sensible multiple test procedure, the first step should be to tailor the selection to the study questions and to the ultimate clinical decision tree. Then evaluation of statistical power performance should come in to play in the next step to fine tune the selected procedure.