Measuring clinical performance: comparison and validity of telephone survey and administrative data

To compare and validate self-reported telephone survey and administrative data for two Health Plan Employer Data and Information Set (HEDIS) performance measures: mammography and diabetic retinal exams.

A telephone survey was administered to approximately 700 women and 600 persons with diabetes randomly chosen from each of two health maintenance organizations (HMOs).

Agreement of survey and administrative data was assessed by using kappa coefficients. Validity measures were assessed by comparing survey and administrative data results to a standard: when the two sources agreed, that was accepted as the standard; when they differed, confirmatory information was sought from medical records to establish the standard. When confirmatory information was not available ranges of estimates consistent with the data were constructed by first assuming that all persons for whom no information was available had received the service and alternately that they had not received the service.

The kappas for mammography were .65 at both HMOs; for retinal exam they were .38 and .40. Sensitivity for both data sources was consistently high. However, specificity was lower for survey (range .44 to .66) than administrative data (.99 to 1.00). The positive predictive value was high for mammography using either data source but differed for retinal exam (survey .69 to .78; administrative data .99 to 1.00).

Administrative and survey data performed consistently in both HMOs. Although administrative data appeared to have greater specificity than survey data the validity and utility of different data sources for performance measurement have only begun to be explored.