An evaluation of statistical approaches for analyzing physician-randomized quality improvement interventions

Health care quality improvement interventions are often evaluated in randomized trials in which individual physicians serve as the unit of randomization. These cluster randomized trials present a unique data structure that consists of many clusters of highly variable size. The appropriate method of analysis for these trials is unknown. We conducted a simulation study to compare several methods for analyzing data which were generated to replicate the structure of our motivating example. We varied the treatment effect size and the distributional assumptions about the random effect. Simulation was used to estimate power, coverage, bias, and mean squared error of full maximum likelihood estimation (MLE), approximate MLE using penalized quasi-likelihood (PQL), generalized estimating equations (GEE), group-bootstrapped logistic regression, and a clustered permutation test. Across all conditions tested, GEE and full MLE performed comparably. Bootstrapped methods were less powerful and had higher mean squared error under conditions of variable cluster size. PQL yielded biased results. The permutation test preserved Type I error rates, but had less power than the other methods considered.