Measurement of inter-rater agreement for transient events using Monte Carlo sampled permutations

In this paper we demonstrate the adverse effect of serially observed data sequences containing transient events on the calculation of Cohen's kappa as an index of inter-rater agreement in the detection of these events. We develop and use a Monte-Carlo-based permutation technique to produce an empiric distribution of kappa in the presence of serial dependence. We find that the empiric confidence intervals for kappa tend to be wider than parametrically derived intervals and in the case of longer event lengths, are markedly so. We evaluate the effect of number and length of events, and further, describe and evaluate three permutation methods which match specific rating situations. Finally, we apply these techniques to the measurement of inter-rater agreement for sleep disordered breathing events, a transient event identified during nocturnal polysomnography, for which traditionally computed confidence intervals for kappa are incorrect.