Assessing the reliability of ordered categorical scales using kappa-type statistics

Area of Science:

• Statistics
• Psychometrics
• Measurement Theory

Background:

• Traditional reliability analysis often relies on single summary coefficients like weighted kappa.
• These single coefficients may not fully capture the nuances of ordered categorical data, especially when an underlying continuum is absent.
• Existing methods have limitations in accurately assessing interobserver reliability with systematic differences.

Purpose of the Study:

• To propose and evaluate a novel method for analyzing the reliability of ordered categorical scales.
• To address the limitations of single summary-weighted kappa coefficients.
• To provide robust estimation methods for reliability coefficients, particularly in interobserver reliability studies.

Main Methods:

• Development of a symmetric matrix of kappa-type coefficients for ordinal scales.
• Application of weighted means of matrix terms when category distances are specified.
• Utilizing the delta-method, bootstrap resampling, and jack-knifing (by subjects and by observer/subject) for precision estimation.
• Empirical comparison of different standard error estimation techniques.

Main Results:

• The symmetric matrix of kappa-type coefficients is proposed as a suitable alternative for ordinal scales without an underlying continuum.
• A weighted mean of matrix terms equates to a weighted kappa with squared weights under specific conditions.
• Standard errors estimated via the delta-method or jack-knifing by subject alone may be overly precise in interobserver reliability studies.

Conclusions:

• The symmetric matrix approach offers a more comprehensive reliability analysis for ordered categorical data.
• Advanced jack-knifing techniques (by observer and subject) provide more reliable precision estimates in interobserver studies.
• The findings advocate for more nuanced reliability assessment beyond single summary coefficients.