Measuring severity of illness: a comparison of interrater reliability among severity methodologies

Methods for measuring illness severity are receiving increasing attention from payers, purchasers, and others interested in the equity and financial incentives of prospective payment systems, as well as from those concerned with the use of mortality rates and other outcomes to measure quality of care. Several methodologies have been proposed for measuring the severity of illness of patients admitted to hospitals. When choosing among the available measures, one characteristic of interest is reliability. In this paper, we present a comparative evaluation of interrater reliability among four severity measures--APACHE II, MedisGroups, Patient Management Categories (PMCs), and Disease Staging Q-Scale--as well as for the Diagnosis Related Groups (DRG) classification system. The results show APACHE II, MedisGroups, and DRGs to be highly reliable, with Inter-Rater Reliability Coefficient (RI) values greater than .8. PMCs and Disease Staging Q-Scale were able to achieve fair-to-good levels of reliability. Results are consistent regardless of which reliability statistics are used.