Comparison of methods to identify outliers observed in health services small area variation studies

Small area variation analysis (SAV) is an established methodology in health services and epidemiological research. The goal is to demonstrate that rates differ across areas, and to explain these differences by differences in physician practice styles or patient characteristics. While the SAV statistics provide an overall variation estimate, they do not provide a statistical means to identify significant outliers. We compared the chi-square (chi2) test with three approaches in determining significant outliers in SAV. We used data from the Canadian Institute for Health Information (CIHI) for Ontario residents discharged between 1989 and 1991. Coronary artery bypass surgery, hysterectomy and hip replacement data were used to compare four statistics in determining outliers: the chi2 test, Swift's approximate bootstrap confidence interval (ABC), Carriere's T2 (T2) with simultaneous confidence intervals (SCI), and Gentleman's normalized scores (GNS). Both the ABC and SCI correct the skewness of the distribution of the adjusted rates. With large data, confidence intervals calculated by the normal or the ABC methods are indistinguishable. The T2 can be applied to also nonbinary events. For binary events, it is asymptotically the same as the chi2. The GNS ranks the rates, but the distribution of these ranks does not differ significantly from that of the adjusted rates. We concluded that when using large data with binary events, there is little advantage in using the ABC, SCI or GNS over the commonly known chi2. The chi2 remains a useful tool in small area variation analysis to 'screen' or flag potential differences beyond chance alone.