Measuring practice systems for chronic illness care: accuracy of self-reports from clinical personnel

Purchasers, plans, and clinical practices involved in quality improvement initiatives are increasingly interested in measuring practice systems, particularly in relation to clinical quality and as part of pay-for-quality initiatives. The validity of self-reports of the use of practice systems was examined.

In 11 medical groups in Minnesota, the Physician Practice Connections Readiness Survey, which was developed on the basis of the concepts and evidence base of the Chronic Care Model, was used to survey office practice personnel about practice systems. Participation rates by medical group ranged from 61% to 94%, with a mean of 76%, yielding surveys from 32 lead physicians and 241 other personnel. Survey results were compared with an on-site audit by trained surveyors.

Overall agreement with the on-site audit ranged from 40.9% to 96.7% among lead physicians and from 33.9% to 81.9% among other personnel. Mean agreement was high for quality improvement (96.7% for lead physicians and 81.9% for other personnel), moderate for clinical information systems (71.2% for lead physicians and 66.0% for others), and low for the use of care management (less than 50% for both groups). Mean positive predictive value ranged from 55.2% to 100% among lead physicians and from 49.6% to 100% among other personnel. Both the presence of systems and the accuracy of reporting varied across medical groups.

The accuracy of self-reports of practice systems varies by type of system being assessed and by type of respondent. Although self-assessment may be useful for quality improvement purposes, self-reported information on clinical practices systems should not be used for accountability purposes, including pay-for-quality efforts or public reporting unless additional documentation is required to ensure fair comparisons.