Does voluntary reporting bias hospital quality rankings?

The Leapfrog Group, a large consortium of private payers, collects voluntary, self-reported quality information from hospitals for several different procedures. Critics argue that voluntary reporting leads to over-representation of high quality hospitals.

We used data from 1169 hospitals participating in the 2008 Leapfrog Survey for six high-risk procedures. We first ranked the hospitals who voluntarily submitted data to Leapfrog using a composite quality measure (a combination of mortality and hospital volume). We then created a similar set of rankings for a national sample of hospitals from the 2006 Nationwide Inpatient Sample (NIS). Next, we created four groups (quartiles) of hospital performance based on the national sample of hospitals. Finally, we compared the Leapfrog hospitals with the national sample of hospitals. If voluntary reporting resulted in bias, we would expect the Leapfrog hospitals to cluster in the top quartiles for each procedure.

The distribution of the Leapfrog hospitals within the NIS performance quartiles varied extensively across procedures. For abdominal aortic aneurysm repair and coronary artery bypass grafting, there was a clustering of hospitals in the two worst quartiles (64% and 65%, respectively). Conversely, for aortic valve replacement, pancreatectomy, percutaneous coronary interventions, and esophagectomy, there were 52%, 56%, 57%, and 61% of hospitals in the top half, respectively.

In the Leapfrog Survey, voluntary reporting did not lead to a systematic over-representation of high quality hospitals. There should be little concern that hospital quality rankings would have significant bias introduced by voluntary participation of hospitals.