A measure of association for ordered categorical data in population-based studies

Ordinal classification scales are commonly used to define a patient's disease status in screening and diagnostic tests such as mammography. Challenges arise in agreement studies when evaluating the association between many raters' classifications of patients' disease or health status when an ordered categorical scale is used. In this paper, we describe a population-based approach and chance-corrected measure of association to evaluate the strength of relationship between multiple raters' ordinal classifications where any number of raters can be accommodated. In contrast to Shrout and Fleiss' intraclass correlation coefficient, the proposed measure of association is invariant with respect to changes in disease prevalence. We demonstrate how unique characteristics of individual raters can be explored using random effects. Simulation studies are conducted to demonstrate the properties of the proposed method under varying assumptions. The methods are applied to two large-scale agreement studies of breast cancer screening and prostate cancer severity.