Some fundamental issues with non-inferiority testing in active controlled trials

In an active controlled non-inferiority trial without a placebo arm, it is often not entirely clear what the primary objective is. In many cases the considered goal is to demonstrate that the experimental treatment preserves at least some fraction of the effect of the active control. The active control effect is a parameter, the value of which is unknown. To test the hypothesis of effect preservation, the classical confidence interval approach requires specification of a non-inferiority margin which is a function of the unknown active control effect. When the margin is estimated, it is also not clear what is the relevant type I error of making a false assertion about preservation of the active control effect. The statistical uncertainty of the estimated margin arguably needs to be incorporated in evaluation of the type I error. In this paper we discuss these fundamental issues. We show that the classical confidence interval approach cannot attain the target type I error exactly since this error varies as the sample size or as the values of the nuisance parameters in the active controlled trial change. In contrast, the preservation tests, as proposed in literature, can attain the target type I error rate exactly, regardless of the sample size and the values of the nuisance parameters, but can do so only at the price of several strong assumptions holding that may not be directly verifiable. One assumption is the constancy condition holding whereby the effect of the active control in the historical trial populations is assumed to carry to the population of the active control trial. When this condition is violated, both the confidence interval approach and the preservation test method may be problematic.