Comparisons of health plan quality: does the sampling strategy affect the results?

Health plan evaluations typically focus on beneficiaries continuously enrolled throughout a calendar year.

To examine whether the sampling strategy used in evaluating health plans affects their assessment.

We completed a cross-sectional analysis of California Medicaid managed care plans aggregating individual level data by health plan.

We calculated the correlation between a health plan's annual age-sex adjusted hospitalization rate for ambulatory care sensitive (ACS) conditions sampling among beneficiaries continuously enrolled throughout the year with all health plan beneficiaries enrolled for a minimum of 1 month. For health plan beneficiaries with a minimum of 1 month of enrollment we calculated hospitalization rates in 2 ways: (1) counting only those ACS hospitalizations while enrolled in the plan and (2) counting any ACS hospitalization throughout the year regardless of whether it was during the period of plan enrollment.

California Medicaid beneficiaries 18-64 years of age in 2001.

Forty-four Medicaid plans representing approximately 750,000 beneficiaries were included. On average, 50% (range, 26-69% across plans) of the beneficiaries were enrolled continuously during the year. Plan rankings based on the ACS hospitalization rates for each of the 3 sampling strategies were variably correlated [Spearman correlations 0.26 (P = 0.086), 0.33 (P = 0.031), and 0.71 (P < 0.0001) for pairwise comparisons]. The agreement among the sampling strategies in labeling a health plan as an outlier was not statistically different than random chance (kappa = 0.069, P = 0.21).

Judgments regarding health plan performance are affected by limiting the evaluation sample to only those beneficiaries continuously enrolled in a health plan throughout the year. Policymakers should consider the goal of health care measurement when selecting a sampling strategy and explicitly acknowledge the bias that might be introduced by a particular approach.