Priority setting in quality assurance: reliability of staff judgments in medical institutions

A structured procedure using the judgments of a representative group of local providers for establishing priorities for quality assurance activity in diverse medical institutions was tested for reliability. Two independent matched teams of phy sicians, nurses, administrators, and other staff in eight separate medical facilities generated 320 topics which encompassed areas where quality assurance efforts would have either considerable or little impact in terms of improving health outcomes within reasonable costs. Concordance of judgment between teams in each facility was determined by analyzing the similarity of topics content, the agreement in scaling the health impact of similar topics generated by both teams independently, and the agreement by one team in scaling the health impact of topics generated by the other team. The findings revealed 44 per cent content agreement on topics independently generated, 93 per cent agreement on dichotomous scaling of similar topics, and 87 per cent agreement on five-point scaling of similar topics. Concordance of judgment by one team in scaling the other team's topics was highly significant (p less than .001). Preliminary analysis of topic content and scaling agreement among different facilities indicated low agreement both on the content areas and on the health impact of similar topics. It is concluded that the judgments of local providers in identifying cost-effective quality assurance priorities is highly relaible in the medical institutions studied.