Task definition, annotated dataset, and supervised natural language processing models for symptom extraction from...

Area of Science:

• Clinical informatics
• Natural Language Processing
• Machine Learning

Background:

• Machine learning (ML) and natural language processing (NLP) show promise for improving information extraction (IE) in electronic medical records (EMRs).
• Clinical adoption of real-time IE tools for patient care is currently limited.
• Development of useful clinical tools requires clinically motivated IE task definitions, annotated clinical datasets, and subtasks like coreference resolution and named entity normalization.

Purpose of the Study:

• To define a clinically motivated symptom extraction task.
• To create and release a comprehensive annotated dataset for symptom extraction.
• To evaluate the performance of ML models on this task.

Main Methods:

• Four annotators labeled symptom mentions in 1108 discharge summaries from two public clinical note datasets.
• Tasks included named entity recognition, coreference resolution, and named entity normalization.
• Two ML models (recurrent network and Transformer-based) were evaluated.

Main Results:

• Over 16,922 symptom mentions were identified, with 11,944 instances after coreference resolution.
• Human annotator performance achieved 92.2% F1 score.
• ML models achieved near-human performance (85.6% F1 for recurrent network, 86.3% F1 for Transformer), extracting complex symptom mentions and generalizing to new data in real time.

Conclusions:

• This work presents the largest publicly released annotated dataset for clinically motivated symptom extraction.
• Developed neural network models demonstrate near-human performance in extracting symptoms from unstructured clinical text in real time.
• The study highlights the feasibility of building clinically applicable IE tools through defined tasks, datasets, and NLP models.