Quality by any other name?: a comparison of three profiling systems for assessing health care quality

Many performance measurement systems are designed to identify differences in the quality provided by health plans or facilities. However, we know little about whether different methods of performance measurement provide similar answers about the quality of care of health care organizations. To examine this question, we used three different measurement approaches to assess quality of care delivered in veteran affairs (VA) facilities.

Medical records for 621 patients at 26 facilities in two VA regions.

We examined agreements in quality conclusions using: focused explicit (38 measures for six conditions/prevention), global explicit (372 measures for 26 conditions/prevention), and structured implicit review physician-rated care (a single global rating of care for three chronic conditions and overall acute, chronic and preventive care). Trained nurse abstractors and physicians reviewed all medical records. Correlations between scores from the three systems were adjusted for measurement error in each using multilevel regression models.

Intercorrelations of scores were generally moderate to high across all three systems, and rose with adjustment for measurement error. Site-level correlations for prevention and diabetes care were particularly high. For example, adjusted for measurement error at the site level, prevention quality was correlated at 0.89 between the implicit and global systems, 0.67 between implicit and focused, and 0.73 between global and focused systems.

We found moderate to high agreement in quality scores across the three profiling systems for most clinical areas, indicating that all three were measuring a similar construct called "quality." Adjusting for measurement error substantially enhanced our ability to identify this underlying construct.