Semantic Krippendorff's α for measuring inter-rater agreement in SNOMED CT coding studies

Semantic interoperability requires consistency in use of terminologies such as SNOMED CT. Inter-rater agreement measurement can be used to quantify this consistency among terminology users. Increasingly, studies of SNOMED CT include inter-rater agreement measures. However, published studies do not consider distance between concepts when calculating the inter-rater agreement measures. In this paper we propose a semantic inter-rater agreement measure for use with SNOMED CT encoded data. A semantic Krippendorff's α measure is implemented using a path-length based difference function. The measure is tested using three different datasets. Results show that the proposed semantic measure is sensitive to seriousness of coding differences whereas a nominal measure is not. The proposed measure reflects the intuition that distance matters when comparing uses of SNOMED CT.