Standardized Annotation of Clinical Narratives with SNOMED CT and FHIR

A semantic framework for manual annotation of clinical narratives using SNOMED CT and FHIR has been created. To test the guideline, two medical annotators independently annotated 20 selected diverse clinical text fragments. Quantitatively, we detect a fair IAA (Cohen's kappa) for concept (κ = 0.32) and a substantial IAA for predicate annotations (κ = 0.78), with 7.97 times faster annotation speed (p<0.001) for predicates and a typical head-tail distribution for concept annotations. Qualitative analysis revealed five factors affecting performance: focus and compliance with annotation guidelines, terminology limitations, contextual information, and tooling support for manual annotation. We conclude that a motivated balance between precise guidelines, automated rule violation checks and pre-annotations, as well as knowledge graph-based IAA, is necessary for a correct semantic annotation of clinical narratives according to international standards.