Small Sample Confidence Bounds on Measures of Agreement of Confusion Matrices

Tau, Lambda, and Kappa are measures developed for the analysis of discrete multivariate data of the type represented by stimulus response confusion matrices. The accuracy with which they may be estimated from small sample confusion matrices is investigated by Monte Carlo methods. Results show that Kappa and Lambda are covered by their confidence intervals more frequently than is Tau. Although most of the intervals are too short for small samples, resulting in fewer than expected hits by the confidence bounds, Kappa and Lambda appear to be quite usable for even very small sample sizes.