A method of developing and weighting explicit process of care criteria for quality assessment

The use of explicit criteria to evaluate how well processes of care conform to accepted standards is a key method of quality assessment. Synthesizing four decades of literature, we devised an inexpensive, 6-step method of developing reliable, content-valid, explicit process criteria. This paper describes the method using a set of congestive heart failure criteria. In step 1 of the Criteria Development Method, criteria are derived from state-of-the-art clinical literature. In step 2, criteria are refined by expert panels. In this study, panelists refined the items by mail in a three-round Delphi process. In step 3, decisions about unit-or differential item weighting are made; we derived differential item weights from the panelists' third-round ratings. Step 4 consists of flagging items which may yield little information, i.e., consensus items of low import, and nonconsensus items. Numeric flags were computed using third-round median ratings and their interquartile ranges. Selection of a scoring method to summarize scores and communicate results is done in step 5. In step 6, chart reviewers are trained, inter-rater reliability is measured, and items with poor reliability are culled. This straightforward developmental method can be used to devise explicit process criteria for use in ambulatory or hospital settings and to evaluate care delivered by different types of providers. The method yields reliable criteria representing accepted standards of current clinical practice. This high content validity is a sine qua non for convergent and predictive validity, both of which must be demonstrated in empirical studies in which the criteria are compared against external yardsticks.