Annotation of epilepsy clinic letters for natural language processing.

Area of Science:

• Medical Informatics
• Natural Language Processing
• Clinical Research

Background:

• Natural Language Processing (NLP) is crucial for extracting structured data from clinical text to support decision-making and research.
• A scarcity of expert-annotated clinical documents hinders the development and validation of NLP tools.
• Synthetic data generation offers a solution to augment limited annotated datasets.

Purpose of the Study:

• To create synthetic clinical documents for training and validating NLP pipelines.
• To develop and assess the performance of the Extraction of Epilepsy Clinical Text version 2 (ExECTv2) NLP pipeline.
• To provide a publicly available resource of annotated epilepsy clinic letters.

Main Methods:

• Generated 200 synthetic clinic letters simulating epilepsy outpatient consultations.
• Employed double annotation by trained clinicians and researchers using the Markup tool and an epilepsy concept list.
• Established a gold standard dataset through annotation review and consensus to validate the ExECTv2 pipeline.

Main Results:

• Achieved an inter-annotator agreement (IAA) F1 score of 0.73 for human annotations.
• The ExECTv2 pipeline achieved a per-item F1 score of 0.87 and a per-letter F1 score of 0.90.
• Demonstrated superior performance of the automated NLP pipeline compared to human annotators.

Conclusions:

• Freely released synthetic letters, annotations, and guidelines for NLP research.
• Highlighted the challenges in clinical text annotation and the necessity of gold standards.
• Confirmed the utility and accuracy of the ExECTv2 NLP pipeline for extracting detailed epilepsy information from unstructured clinical text.