Using Internal Inconsistency To Identify Unreliable Ratings

This paper describes the development and testing of an internal inconsistency scale to help identify unreliable clinical ratings. The scale consists of pairs of logically inconsistent items which have been chosen and scored using a combination of rational and statistical procedures. Evidence is presented to indicate that the scale measures rater inconsistencies rather than patient "inconsistencies," that using inconsistency scores to eliminate unreliable ratings improves factor score reliabilities, and that increasing the number of items for random assignment of possible rating values systematically increases the inconsistency score. Although the inconsistency scale was constructed specifically for the Missouri Inpatient Behavior Scale (MIBS), the methodology is potentially applicable to other rating scales.