Reliability of a utilization review instrument in a large field study

One important question for a utilization management program is whether the utilization review instrument is consistent or stable when used on many occasions by the same abstractor (intrarater reliability) or by several abstractors (inter-rater reliability). As part of a nationwide study of inappropriate utilization of inpatient services by the Department of Veterans Affairs, we conducted a thorough investigation of the inter-rater reliability of a widely used utilization review instrument by 27 nurse abstractors. All abstractors were extensively trained, both by the developers of the instrument and by use of practice medical records. A standard protocol for resolving questions was implemented, with immediate communication of decisions to abstractors. The results of three reliability assessments, conducted immediately after formal training, after several weeks of reviewing practice records, and midway through review of the study records, demonstrated good to excellent reliability, both when comparing the nurse abstractors with a physician gold standard and among themselves. Therefore, with appropriate training and monitoring, utilization management programs in large hospitals, multihospital systems, and other health care organizations needing to examine inpatient utilization should feel confident that they can achieve reviews that would be in close agreement with physician and other nurse abstractors. Such confidence should increase the acceptability of utilization management programs.