Clinical concept extraction using transformers.

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning
• Biomedical Informatics

Background:

• Clinical concept extraction is crucial for processing medical text.
• Existing methods may not fully capture the nuances of clinical language.
• Transformer-based models offer advanced capabilities for NLP tasks.

Purpose of the Study:

• To evaluate transformer-based models for clinical concept extraction.
• To develop and release an open-source package of pretrained clinical NLP models.
• To enhance downstream NLP applications in the medical domain.

Main Methods:

• Systematic exploration of four transformer architectures (BERT, RoBERTa, ALBERT, ELECTRA).
• Utilized three public datasets from i2b2 and n2c2 challenges.
• Compared transformer models (general and clinical pretrained) against a LSTM-CRFs baseline.

Main Results:

• RoBERTa-MIMIC achieved state-of-the-art performance with F1-scores up to 0.8994.
• Significant F1-score improvements of ~4-6% over the baseline LSTM-CRFs model.
• Demonstrated the superior efficiency of transformer models for clinical concept extraction.

Conclusions:

• Transformer-based models are highly effective for clinical concept extraction.
• The developed open-source package facilitates clinical NLP tasks.
• The findings and tools can advance various medical domain applications.