Weighted specific-category kappa measure of interobserver agreement

When two observers classify a sample of items using the same categorical scale, and when different disagreements are differentially weighted, the weighted Kappa (Kw) by Cohen may serve as a measure of interobserver agreement. We propose a Kappa-based weighted measure (K(ws)) of agreement on some specific category s, with Kw being a weighted average of all K(ws)s. Therefore, while Cohen's Kw is a summary measure of the overall agreement, the proposed K(ws) provides a measure of the extent to which the observers agree on the specific categories, with both measures being suitable for ordinal categories because of the weights being used. Statistical inferences for K(ws) and its unweighted counterpart are also discussed. A numerical example is provided.