Concordance between ratings using different scales for the same variable

Qualitative variables are commonly measured by means of different types of rating scales that produce ordered categorical data. The number of response categories in a scale can be chosen arbitrarily according to different operational definitions of the attribute to be measured. In the present paper classical measures based on the difference between the probabilities of concordant and discordant pairs of classifications were compared with a novel approach for assessment of order consistency between rating scales. In the new approach, illustrated by an assessment of pain, the concordant and discordant pairs are related to the pattern of total order consistency, irrespective of the scaling and the categorical distributions. The level of order consistency between a visual analogue scale (VAS) and a discrete scale for the same variable was evaluated. The non-linear properties of VAS assessments were demonstrated. The inter-scale consistency between the discrete scale and different approaches to grouping VAS responses into discrete scales showed that equidistant rescaling of VAS assessments resulted in an inter-scale bias.