BioBERT: a pre-trained biomedical language representation model for biomedical text mining.

Area of Science:

• Biomedical informatics
• Natural Language Processing (NLP)
• Deep Learning

Background:

• The rapid growth of biomedical literature necessitates advanced text mining techniques.
• Standard NLP models often struggle with biomedical text due to domain-specific language.
• Deep learning has advanced biomedical text mining, but domain adaptation is crucial.

Purpose of the Study:

• To adapt the pre-trained language model BERT for the biomedical domain.
• To develop a domain-specific language representation model for biomedical text mining.
• To evaluate the performance of the adapted model on various biomedical NLP tasks.

Main Methods:

• Introduced BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), a BERT model pre-trained on large-scale biomedical corpora.
• Evaluated BioBERT on biomedical named entity recognition, relation extraction, and question answering tasks.
• Compared BioBERT's performance against the original BERT and previous state-of-the-art models.

Main Results:

• BioBERT significantly outperformed BERT and prior state-of-the-art models across multiple biomedical text mining tasks.
• Achieved notable improvements: 0.62% F1 score for named entity recognition, 2.80% F1 score for relation extraction, and 12.24% MRR for question answering.
• Demonstrated that pre-training on biomedical corpora enhances BERT's understanding of complex biomedical texts.

Conclusions:

• Pre-training BERT on biomedical corpora, creating BioBERT, is an effective strategy for improving biomedical text mining.
• BioBERT offers superior performance for key tasks, advancing information extraction from biomedical literature.
• The pre-trained weights and fine-tuning code for BioBERT are publicly available to facilitate research.