Group judgments of appropriateness: the effect of panel composition

The current interest in the development of practice guidelines raises an important question about the effect of expert panel composition on the outcome of the guideline development process. We compared the ratings of appropriateness of indications for carotid endarterectomy produced by two panels: an all-surgical panel and a "balanced" panel composed of four surgeons, two neurologists, and one specialist each from family practice, internal medicine, and radiology. We then compared the effect of the two sets of ratings when used to evaluate 1302 patients who had undergone carotid endarterectomy. The all-surgical panel found more indications "appropriate" (24 versus 14%) and fewer indications "inappropriate" (61 versus 70%) than the balanced panel (p less than 0.01). The all-surgical panel also more often reached agreement. When ratings were applied to patients, 70% were appropriate by the all-surgical criteria versus 38% by the balanced panel, while 19% of the operations were inappropriate by all-surgical criteria, versus 31% by the balanced panel ratings. However, the percentage of procedures judged "inappropriate with agreement" was 15% for all-surgical and 16% for the balanced panel. We conclude that the all-surgical panel was more likely to favor operative treatment for carotid disease than the multispecialty panel, but that concensus judgments of inappropriateness by the two panels were very similar.