Separation of systematic and random differences in ordinal rating scales

We introduce a new statistical method, which separates and measures different types of variability between paired ordered categorical measurements. The key to the separation is a two-way augmented ranking approach of observations in a contingency table. It means that cases classified in a specific category by one rater will be internally ranked according to the classifications from the other. This enables us to extract the component of interobserver variation which is not systematic. The variance of the rank differences between judgements is a suitable measure of this interrater variability, which we characterize as random. The empirical measure of random interjudge disagreement, which lies between 0 and 1, is called the relative rank variance and is an estimate of a parameter defined on the multinomial probability distribution in the contingency table. The systematic differences are determined by the marginals and described by two empirical measures, relative position and relative concentration; both measures lie between -1 and 1. Our method is applied to data sets from a reliability study of two clinical rating scales for assessing hydrocephalus and subarachnoid haemorrhage.