Constructing a corpus of hematologic pathology notes for the fine-tuning of BERT models for named entity recognition

Microscopic, immunologic, and chemical testing play a major role in the diagnostic process of hematologic cancer patients. Pathologists record the complex results of these tests in highly descriptive, free-text clinical notes. As unstructured text is not searchable, the relevant information in the notes is difficult to access for research and further analysis.

The objective was the construction of a corpus of annotated hemato-oncological pathology notes as part of the development of an automated system for the extraction of structured data from clinical notes. The corpus was evaluated through the fine-tuning and assessment of a Bidirectional Encoder Representations from Transformers (BERT) model in the context of Named Entity Recognition (NER).

We developed guidelines and an annotation scheme that capture relevant information in our hematology reports, such as diagnoses, immunohistochemistry findings, proliferation rate, presence of light chain restriction, and mutations. We annotated 110 reports twice in several rounds and measured the inter-annotator-agreement (IAA) using the F1-score. Following each round, we discussed inconsistencies and updated the annotation guidelines and scheme accordingly. We mapped the terms of our annotation strategy to systematized nomenclature of medicine clinical terms (SNOMED CT) concepts to permit translation into other languages. To validate our strategy, we fine-tuned a pre-trained BERT-model for a NER task using 240 annotated texts.

The final annotation scheme consists of nine entity types and ten attribute types. During the annotation process, we observed an improvement in the F1-score for the IAA from 0.61 to 0.85 (close match) and from 0.70 to 0.91 (relaxed match). The model achieved an overall F1-score of 0.88 for the classification of entities and attributes in the validation experiment.

We developed a novel and extensive annotation scheme and guidelines for the annotation of hemato-oncological pathology notes. The annotations facilitate experiments with state-of-the-art NER models which yielded satisfactory results in identifying entities and attributes.