Kappa and attenuation of the odds ratio

This paper explores the relation between reliability and validity for binary variables. An equation is first derived for expressing the kappa coefficient of reliability as a function of true prevalence, sensitivity, specificity, and the degree of correlation of errors within true positives and within true negatives. It is then shown that under certain circumstances kappa is closely related to an index of the validity of the odds ratio as a means for quantifying an association between an imperfectly classified binary factor and membership in two subpopulations. This index is defined as the ratio of the observed odds ratio minus one to the true odds ratio minus one. Numerical results indicate that, when error rates are high, kappa can be of smaller magnitude than this index of validity. When the value of kappa reflects substantial correlation of errors, however, the value of kappa typically exceeds that of the index of validity. Correlated errors among the true negatives are found to affect the correspondence between kappa and the index of validity rather differently than do correlated errors among the true positives. Although kappa often exceeds the index of validity, use of kappa to correct the odds ratio for attenuation owing to nondifferential misclassification generally leads to undercorrection rather than overcorrection. Consequently, the relation between kappa and the index of the validity of the odds ratio may have practical value for reducing bias in the estimation of exposure-disease associations in epidemiologic research.