Reliability of multi-category rating scales

Area of Science:

• Educational Psychology
• Measurement and Statistics

Background:

• Multi-category scales are increasingly used for monitoring Individualized Education Program (IEP) goals, classroom rules, and Behavior Improvement Plans (BIPs).
• Assessing the inter-rater reliability of these scales is crucial but understudied, especially compared to traditional data counting methods.

Purpose of the Study:

• To examine the performance of nine different reliability indices when applied to multi-category scales.
• To investigate how scale gradation (number of categories) and data grouping affect reliability index values.

Main Methods:

• A simulation study was conducted using quasi-continuous data (1-30) transformed into six multi-category scales (2, 3, 5, 7, 10, and 15 points).
• Nine distinct reliability indices were applied to these scales to evaluate their behavior and performance.

Main Results:

• Each reliability index demonstrated unique performance characteristics, necessitating individualized interpretation.
• No single 'best' reliability index was identified; most indices proved to be scale-dependent.
• Index values changed significantly when categories were collapsed, highlighting the impact of scale structure.

Conclusions:

• Current reliability indices are not universally applicable to multi-category ordinal scales.
• New guidelines are required for selecting and interpreting reliability measures for these educational assessment tools.
• Further research is needed to establish optimal methods for assessing inter-rater reliability with ordinal scales.