Physician Cost Profiling-Reliability and Risk of Misclassification: Detailed Methodology and Sensitivity Analyses

This article describes the methods and sensitivity analyses used by the authors in an article published in the New England Journal of Medicine. Purchasers are experimenting with a variety of approaches to control health care costs, including limiting network contracts to lower-cost physicians and offering patients differential copayments to encourage them to visit "high-performance" (i.e., higher-quality, lower-cost) physicians. These approaches require a method for analyzing physicians' costs and a classification system for determining which physicians have lower relative costs. There has been little analysis of the reliability of such methods. Reliability is determined by three factors: the number of observations, the variation between physicians in their use of resources, and random variation in the scores. A study of claims data from four Massachusetts health plans demonstrates that, according to the current methods of physician cost profiling, the majority of physicians did not have cost profiles that met common reliability thresholds and, importantly, reliability varied significantly by specialty. Low reliability results in a substantial chance that a given physician will be misclassified as lower-cost when he or she is not, or vice versa. Such findings raise concerns about the use of cost profiling tools and the utility of their results. It also explains the relationship between reliability measurement and misclassification for physician quality and cost measures in health care. It provides details and a practical method to calculate reliability and misclassification from the data typically available to health plans. This article builds on other RAND work on reliability and misclassification and has two main goals. First, it can serve as a tutorial for measuring reliability and misclassification. Second, it will describe the likelihood of misclassification in a situation not addressed in our prior work in which physicians are categorized using statistical testing. For any newly proposed system, the methods presented here should enable an evaluator to calculate the reliabilities and, consequently, the misclassification probabilities. It is our hope that knowing these misclassification probabilities will increase transparency about profiling methods and stimulate an informed debate about the costs and benefits of alternative profiling systems.