Health plan report cards: exploring differences in plan ratings

Despite the considerable attention paid to the collection of data and the publication of health plan report cards, there is no available study on the comparability of published report cards. Ratings from seven health plan report cards publicly available in fall 1996 were compared--including those produced by major periodicals, a large national employer, a nonprofit consumer group, a health maintenance organization accreditation agency, and a consortium of employers.

Dimensions of plan performance common to the seven report cards were identified. Spearman rank correlation coefficients were computed for each pair of report cards for each of the three dimensions that were evaluated. COMPARABILITY OF REPORT CARDS: Although plan ratings tended to be positively correlated as hypothesized, the magnitude of the estimated correlation coefficients varied. For example, the estimated correlation coefficient between two periodicals' overall plan ratings was 0.48. The ranges of estimated correlations were 0.18-0.70 for preventive care (among four report cards) and 0.19-0.73 for enrollee satisfaction (among three report cards).

Discrepancies in ratings may reflect methodologic issues pertaining to the sample of health plans used, plan performance measures included, and the processes by which individual measures were aggregated to construct indices and ratings. Health plan report cards may be sending mixed signals to consumers. These inconsistencies may explain why focus group studies have found that despite the widespread indication that plan performance measures would be useful, relatively few of those who had seen such information report using it in making their plan choice. Future efforts to evaluate health plans should clearly identify assumptions, methods, normative judgments, and limitations.