Confidence intervals for the difference between independent binomial proportions: comparison using a graphical...

This paper uses graphical methods to illustrate and compare the coverage properties of a number of methods for calculating confidence intervals for the difference between two independent binomial proportions. We investigate both small-sample and large-sample properties of both two-sided and one-sided coverage, with an emphasis on asymptotic methods. In terms of aligning the smoothed coverage probability surface with the nominal confidence level, we find that the score-based methods on the whole have the best two-sided coverage, although they have slight deficiencies for confidence levels of 90% or lower. For an easily taught, hand-calculated method, the Brown-Li 'Jeffreys' method appears to perform reasonably well, and in most situations, it has better one-sided coverage than the widely recommended alternatives. In general, we find that the one-sided properties of many of the available methods are surprisingly poor. In fact, almost none of the existing asymptotic methods achieve equal coverage on both sides of the interval, even with large sample sizes, and consequently if used as a non-inferiority test, the type I error rate (which is equal to the one-sided non-coverage probability) can be inflated. The only exception is the Gart-Nam 'skewness-corrected' method, which we express using modified notation in order to include a bias correction for improved small-sample performance, and an optional continuity correction for those seeking more conservative coverage. Using a weighted average of two complementary methods, we also define a new hybrid method that almost matches the performance of the Gart-Nam interval.