Separation of systematic and random differences in ordinal rating scales

Area of Science:

• Statistics
• Biostatistics
• Medical Informatics

Background:

• Assessing agreement between raters using ordered categorical data is crucial in clinical research.
• Existing methods may not adequately distinguish between systematic and random interrater variability.
• Reliability studies require robust statistical tools to analyze measurement consistency.

Purpose of the Study:

• To introduce a novel statistical method for separating and measuring distinct types of variability in paired ordered categorical measurements.
• To provide empirical measures for random (non-systematic) and systematic interrater variability.
• To apply the developed method to assess the reliability of clinical rating scales.

Main Methods:

• A two-way augmented ranking approach within a contingency table framework.
• Internal ranking of observations based on classifications from multiple raters.
• Calculation of relative rank variance for random interrater variability (0-1).
• Determination of systematic differences using relative position and relative concentration measures (-1 to 1).

Main Results:

• The method successfully separates and quantifies random and systematic variability components.
• Relative rank variance estimates random interjudge disagreement.
• Relative position and concentration measures capture systematic differences in ratings.
• Application to clinical rating scales for hydrocephalus and subarachnoid hemorrhage demonstrated method utility.

Conclusions:

• The proposed statistical method offers a comprehensive approach to analyzing variability in paired ordered categorical data.
• It provides valuable insights into both random and systematic sources of interrater disagreement.
• This method can improve the reliability assessment of clinical rating scales and other measurement tools.