A note on interval estimation of kappa in a series of 2 x 2 tables

When there are confounders in reliability studies, failing to stratify data to account for these confounding effects may produce a misleading estimate of the interrater agreement. In this paper, we focus discussion on interval estimation of kappa for measuring agreement between two raters with stratified data. Using Monte Carlo simulation, we compare four asymptotic interval estimators of kappa for stratified data: estimator (1), a weighted average of the stratum-specific kappa estimates with weights equal to the inverse of the estimated asymptotic variances of these estimates; the two estimators, (2) and (3), with use of the logarithmic and the square root transformations, respectively, and a principle similar as used in estimator (1); estimator (4), a weighted average of the stratum-specific kappa estimates with weights equal to stratum sizes. We find that while the coverage probability of the first three interval estimators (1)-(3) can often be less than the desired confidence level, the fourth interval estimator (4) consistently performs well in all the situations considered here. We further find that when the underlying kappa is moderate (0.30