Limitations of the facility-specific standardized mortality ratio for profiling health care quality in dialysis

Health care quality is assessed by profiling measures of care and/or health outcomes. However, such tools to measure outcome as standardized mortality ratio (SMR) are often used without thorough validation of their strengths and limitations. Our study compared the dialysis facility-specific SMR and SMR-based rating using different statistical methods and followed them over time. All Fresenius Medical Care, North America dialysis facilities (n = 377) that contributed patient data from 1993 to 1995 (>103,500 patient-years) were included. Four distinct statistical methods (US Renal Data System [USRDS], Poisson, logistic, and Cox regression) were used to compute facility-specific SMRs and rank and classify facilities. The analysis compared the SMR and SMR-based rating of dialysis facilities between SMR method and over time. Different methods produced statistically significant differences in SMR distribution (P < 0.05). The USRDS method produced SMR values that decreased over time (P < 0.001). Based on 90% confidence intervals to determine outliers, the SMR-dependent ranking of dialysis facilities varied by method (P < 0.001). SMR-based ranking was stable over time except for the USRDS method (P < 0.001). Contingency table analysis showed up to a 33% total misclassification rate between SMR methods when ranking facilities. The facility-specific SMR and SMR-based ranking are both sensitive to statistical technique. Because the SMR yields different results in a year and over time and because there is no demonstrable gold standard, conclusions based on any one technique are unstable and unreliable. Regulatory monitoring, actions, and/or performance awards should be avoided based on this measure. However, a facility-specific SMR estimated in any valid way may be useful as an epidemiological research tool.