Fine-Tuning Bidirectional Encoder Representations From Transformers (BERT)-Based Models on Large-Scale Electronic...

Area of Science:

• Natural Language Processing (NLP)
• Biomedical Informatics
• Clinical Informatics

Background:

• Bidirectional Encoder Representations from Transformers (BERT) models excel in NLP tasks.
• Limited research has applied BERT to biomedical and clinical entity normalization.
• Entity normalization is crucial for organizing and understanding biomedical and clinical data.

Purpose of the Study:

• To evaluate BERT-based models for biomedical and clinical entity normalization.
• To assess the impact of training data domains on BERT model performance.
• To compare novel BERT models against established normalization systems.

Main Methods:

• Fine-tuned BioBERT on 1.5 million unlabeled electronic health record (EHR) notes to create EhrBERT.
• Further fine-tuned EhrBERT, BioBERT, and BERT on three annotated corpora (MADE 1.0, NCBI disease, CDR).
• Compared performance against MetaMap and DNorm.

Main Results:

• EhrBERT achieved 40.95% F1 on the MADE 1.0 corpus, outperforming MetaMap.
• EhrBERT improved F1 scores on the NCBI disease corpus (90.35%) and CDR corpus (93.82%) compared to DNorm.
• EhrBERT outperformed BioBERT and BERT on the MADE 1.0 and CDR corpora.

Conclusions:

• BERT-based models achieve state-of-the-art performance in biomedical and clinical entity normalization.
• These models demonstrate adaptability for normalizing diverse named entities.
• EhrBERT shows significant potential for improving clinical data processing and analysis.