Rates of false flagging due to statistical artifact in CMS evaluations of transplant programs: results of a...

The recent CMS conditions of participation are based on risk-adjusted models produced by the Scientific Registry for Transplant Recipients (SRTR). The accuracy of these models in identifying poor-performing centers is unknown. In this stochastic simulation study, 1-year mortality outcomes were simulated in virtual transplant centers, and used to flag centers according to the methods used by CMS, evaluating nine overlapping 2.5-year periods of simulated data. In a simulation where all centers had the same underlying risk, 10.2% were falsely flagged at least once during the 4.5 years of simulated evaluations. The probability of false-positive flagging was lowest in low-volume centers (2.5%) and highest in high-volume centers (16.2%). In another simulation where 5% of centers were assigned twofold risk ("poor-performing centers"), only 32% of poor-performing centers were correctly flagged. In a final simulation where each center was assigned a unique mortality risk, 94% of flagged centers had greater-than-median risk, but only 32% of flagged centers were among the 5% with highest risk. Even after disregarding known covariate limitations to the risk adjustment models, statistical noise alone leads to spurious flagging of many adequately-performing transplant centers, yet the methods used by CMS fail to flag most centers with true elevated risk.