Appropriateness of care. A comparison of global and outcome methods to set standards

The RAND-UCLA Health Services Utilization Study previously analyzed the appropriateness of use of carotid endarterectomy based on a literature review and global expert judgments. In this study, for 45 of the same clinical indications used in the RAND-UCLA Study, the authors compare the appropriateness judgments based on the global judgment method to appropriateness ratings based on probability estimates of specific outcomes that were provided by the same panel of experts. The authors asked these experts to estimate, for each clinical indication, the likelihood of important medical outcomes (i.e. stroke within a year) in the presence and absence of endarterectomy. Using decision analysis, the appropriateness of endarterectomy for these 45 indications was then calculated. For only two of the eight physicians were the Spearman rank-order correlations between these two methods of judging appropriateness significant and positive. (Correlations for the eight physicians ranged from 0.45 to -0.38). This result was produced by: 1) the tendency of the experts to estimate relatively poor outcomes for seriously ill patients whether or not endarterectomy was performed; and 2) a far less consistent effect of clinical factors on outcome estimates than on global judgments. Better methods for incorporating probability estimates into a global rating process must be developed. The authors found excellent agreement between the panelists' relative outcome estimates for common endarterectomy indications and the observed stroke rate for these same indications, suggesting that one promising method is to use specific outcome data to "anchor" expert judgments.