A proposed prototype for identifying and correcting sources of measurement error in classification systems

Because many raters are generally involved in the implementation of a patient classification system, interrater reliability is always a concern in the development and use of such a system. In this article, a case example is used to demonstrate a prototype for identifying measurement error introduced at each step in the classification process (assessment, creating summary item responses, and use of these responses for categorization) and to illustrate how this identification may lead to error reduction strategies. The methods of analyses included percent agreement, Kappa, and visual inspection of contingency tables displaying interrater responses to assessment items, summary items, and the placement category. The extent to which raters followed instructions was analyzed by comparing their responses with computer-generated responses across the classification steps. In addition, raters were interviewed regarding their use of the system.