Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize...

Area of Science:

• Natural Language Processing
• Machine Learning
• Bioinformatics

Background:

• Large-scale neural language models enhance transfer learning in NLP.
• Transformer-based models like BERT improve information extraction but struggle in data-scarce domains.
• Training domain-specific BERT models is challenging due to limited high-quality, large-scale databases.

Purpose of the Study:

• To address the challenge of training high-performance BERT models in domains with limited data.
• To investigate the effectiveness of oversampling domain-specific corpora for pretraining.
• To develop and evaluate a novel pretraining method for domain-specific BERT models.

Main Methods:

• Simultaneous pretraining of models using oversampled knowledge from distinct domains.
• Development of English biomedical BERT from a small corpus.
• Creation of Japanese medical BERT from a small corpus.
• Enhanced biomedical BERT pretraining using balanced PubMed abstracts.

Main Results:

• English BERT achieved practical performance on the Biomedical Language Understanding Evaluation (BLUE) benchmark.
• Proposed method outperformed conventional methods for biomedical corpora of similar sizes.
• Japanese medical BERT surpassed conventional models on most medical tasks.
• Enhanced biomedical BERT showed superior clinical and biomedical scores on the BLUE benchmark.

Conclusions:

• Balanced pretraining with oversampled, task-appropriate corpora enables high-performance BERT model construction.
• The proposed method offers a viable solution for developing effective domain-specific language models.
• This approach is particularly beneficial for specialized fields with data limitations.