Estimating the risk of rare complications: is the 'rule of three' good enough?

THE CLINICAL PROBLEM: If a surgeon has performed a particular operation on n consecutive patients without major complications, what is the long-term risk of major complications after performing many more such operations? Examples of such operations are endoscopic cholecystectomy, nephrectomy and sympathectomy. THE STATISTICAL PROBLEM AND SOLUTIONS: This general problem has exercised the minds of theoretical statisticians for more than 80 years. They agree only that the long-term risk is best expressed as the upper bound of a 95% confidence interval. We consider many proposed solutions, from those that involve complex statistical theory to the empirical 'rule of three', popular among clinicians, in which the percentage risk is given by the formula 100 x (3/n). OUR CONCLUSIONS: The 'rule of three' grossly underestimates the future risks and can be applied only when the initial complication rate is zero (that is, 0/n). If the initial complication rate is greater than zero, then no simple 'rule' suffices. We give the results of applying the more popular statistical models, including their coverage. The 'exact' Clopper-Pearson interval has wider coverage across all proportions than its nominal 95%, and is, thus, too conservative. The Wilson score confidence interval gives about 95% coverage on average overall population proportions, except very small ones, so we prefer it to the Clopper-Pearson method. Unlike all the other intervals, Bayesian intervals with uniform priors yield exactly 95% coverage at any observed proportion. Thus, we strongly recommend Bayesian intervals and provide free software for executing them.