Meaningful variation in performance: a systematic literature review

Recommendations for directing quality improvement initiatives at particular levels (eg, patients, physicians, provider groups) have been made on the basis of empirical components of variance analyses of performance.

To review the literature on use of multilevel analyses of variability in quality.

Systematic literature review of English-language articles (n = 39) examining variability and reliability of performance measures in Medline using PubMed (1949-November 2008).

Variation was most commonly assessed at facility (eg, hospital, medical center) (n = 19) and physician (n = 18) levels; most articles reported variability as the proportion of total variation attributable to given levels (n = 22). Proportions of variability explained by aggregated levels were generally low (eg, <19% for physicians), and numerous authors concluded that the proportion of variability at a specific level did not justify targeting quality interventions to that level. Few articles based their recommendations on absolute differences among physicians, hospitals, or other levels. Seven of 12 articles that assessed reliability found that reliability was poor at the physician or hospital level due to low proportional variability and small sample sizes per unit, and cautioned that public reporting or incentives based on these measures may be inappropriate.

The proportion of variability at levels higher than patients is often found to be "low." Although low proportional variability may lead to poor measurement reliability, a number of authors further suggested that it also indicates a lack of potential for quality improvement. Few studies provided additional information to help determine whether variation was, nevertheless, clinically meaningful.