Weighted least-squares approach for comparing correlated kappa

Area of Science:

• Medical Statistics
• Biostatistics
• Health Services Research

Background:

• Assessing agreement between raters or instruments is crucial in medical research.
• The kappa coefficient is a widely used metric for evaluating agreement in categorical data.
• Existing methods for comparing dependent kappa coefficients can be complex.

Purpose of the Study:

• To present a weighted least-squares (WLS) approach for testing the equality of two dependent kappa coefficients.
• To demonstrate the application of SAS PROC CATMOD for analyzing dependent kappa coefficients.
• To provide a practical method for evaluating inter-rater or inter-instrument agreement in biomedical studies.

Main Methods:

• Utilized the weighted least-squares (WLS) approach, accounting for correlations between estimated kappa statistics.
• Employed SAS PROC CATMOD to test equality of dependent Cohen's kappa and intraclass kappa for nominal ratings.
• Extended the methodology to compare dependent Cohen's kappa and weighted kappa for ordinal ratings.

Main Results:

• The WLS approach provides a statistically sound method for comparing dependent kappa coefficients.
• SAS PROC CATMOD effectively implements the WLS method for both nominal and ordinal data.
• The methodology was successfully illustrated using data from three biomedical studies.

Conclusions:

• The WLS approach offers a valuable tool for researchers assessing agreement in dependent categorical data.
• SAS software facilitates the practical application of this method in medical research.
• This technique enhances the ability to rigorously evaluate reliability in diagnostic and assessment tools.