Utility and pitfalls of some statistical methods in active controlled clinical trials

Increasingly often, the study objective in an active controlled clinical trial without a placebo arm is to show that a new treatment is no less effective than the active control treatment within some noninferiority range. Two issues behind this objective are that of whether the new treatment is efficacious relative to a putative placebo and that of whether the new treatment preserves a certain fraction of effect of the active control. To address these issues, two types of statistical analysis methods are employed in recent pharmaceutical applications. In one type of method, a noninferiority margin is determined, and then the relative effect of the new treatment versus the control is compared against the margin to test noninferiority and the efficacy of the new treatment. In the other type of method, a synthetic statistic is constructed to directly estimate or test the effect of the new treatment relative to the putative placebo without resorting to noninferiority argument. Preservation of control effect can also be estimated and tested. These methods carry some crucial assumptions. The effect of active control is often estimated from a collection of historical placebo controlled trials using the random effects modeling of DerSimonian and Laird. In this work we find that statistical validity of the latter method rests highly on the assumptions that control effect is not reduced in the current active controlled trial population compared to the historical trials and that a normal approximation is appropriate in the random effects modeling. This type of method is very sensitive to departure from these assumptions. In contrast, the former method is ultraconservative in terms of type I error when the assumptions are met and can be anticonservative when control effect is substantially less in the active controlled trial than estimated from the historical placebo controlled trials.