Homogeneity of kappa statistics in multiple samples.

Area of Science:

• Statistics
• Biostatistics
• Psychometrics

Background:

• Intra-observer agreement measurement for categorical data is crucial.
• Cohen's kappa statistic is a foundational measure for nominal scales.
• Existing methods address various agreement scenarios, including multiple raters and classification schemes.

Purpose of the Study:

• To provide a comprehensive overview of statistical methods for intra-observer agreement.
• To introduce and facilitate the use of kappa statistics in research.
• To offer a computational tool for assessing agreement homogeneity.

Main Methods:

• Review of statistical literature on agreement measures.
• Development of procedures for dichotomous and polytomous classification schemes.
• Implementation of inference procedures for homogeneity testing of kappa statistics.
• Provision of executable FORTRAN code for kappa homogeneity testing (kappa(h)).

Main Results:

• Established Cohen's kappa as a key metric for categorical data agreement.
• Developed advanced methods for multi-rater and multi-category agreement assessment.
• Introduced statistical tests for homogeneity of kappa values across independent studies.
• Provided a practical FORTRAN program for kappa homogeneity analysis.

Conclusions:

• Cohen's kappa and its extensions are vital for evaluating measurement quality.
• The developed methods and FORTRAN code enhance the statistical rigor of agreement studies.
• The kappa homogeneity test is valuable for multi-site or multi-study comparisons.