Meaningful variation in performance: a systematic literature review

Area of Science:

• Health Services Research
• Quality Improvement Science
• Statistical Modeling in Healthcare

Background:

• Quality improvement initiatives often rely on performance variation analyses.
• Recommendations for targeting interventions are based on empirical data at patient, physician, and provider group levels.

Purpose of the Study:

• To systematically review the literature on the application of multilevel analyses to understand variability in healthcare quality.
• To evaluate how variability in performance measures is assessed and interpreted in quality improvement research.

Main Methods:

• A systematic literature review of 39 English-language articles published between 1949 and November 2008.
• Searches were conducted in Medline using PubMed, focusing on variability and reliability of performance measures.

Main Results:

• Variation was most frequently analyzed at the facility (hospital, medical center) and physician levels.
• Proportions of total variation attributed to higher levels (e.g., physicians) were generally low (<19%), leading many authors to question the justification for targeting interventions at these levels.
• Poor measurement reliability was found at the physician or hospital level in several studies, cautioning against public reporting or incentives based on these measures.

Conclusions:

• Low proportional variability at levels above patients is common.
• Low variability may indicate poor measurement reliability and potentially limited opportunities for quality improvement at these levels.
• Few studies provided sufficient data to determine the clinical meaningfulness of observed performance variations.