Intraclass correlation for two-by-two tables under three sampling designs

Several sampling designs for assessing agreement between two binary classifications on each of n subjects lead to data arrayed in a four-fold table. Following Kraemer's (1979, Psychometrika 44, 461-472) approach, population models are described for binary data analogous to quantitative data models for a one-way random design, a two-way mixed design, and a two-way random design. For each of these models, parameters representing intraclass correlation are defined, and two estimators are proposed, one from constructing ANOVA-type tables for binary data, and one by the method of maximum likelihood. The maximum likelihood estimator of intraclass correlation for the two-way mixed design is the same as the phi coefficient (Chedzoy, 1985, in Encyclopedia of Statistical Sciences, Vol. 6, New York: Wiley). For moderately large samples, the ANOVA estimator for the two-way random design approximates Cohen's (1960, Psychological Measurement 20, 37-46) kappa statistic. Comparisons among the estimators indicate very little difference in values for tables with marginal symmetry. Differences among the estimators increase with increasing marginal asymmetry, and with average prevalence approaching .50.