Comparison of validity of assessment methods using indices of adjusted agreement

For comparing the validity of rating methods, the adjusted kappa (S coefficient) and Yule's Y index are better than Cohen's kappa which is affected by marginal probabilities. We consider a validity study in which a subject is assessed as exposed or not-exposed by two competing rating methods and the gold standard. We are interested in one of the methods, which is closer in agreement with the gold standard. We present statistical methods taking correlations into account for comparing the validity of the rating methods using S coefficient and Y index. We show how the S coefficient and Yule's Y index are related to sensitivity and specificity. In comparing the two rating methods, the preference is clear when the inference is the same for both S and Y. If the inference using S differs from that using Y, then it is not obvious how to decide a preference. This may occur when one rating method is better than the other in sensitivity but not in specificity. Numerical examples for comparing asbestos-exposure assessment methods are illustrated.