Bioformer: an efficient transformer language model for biomedical text mining.

Area of Science:

• Computational Biology
• Natural Language Processing
• Bioinformatics

Background:

• Bidirectional Encoder Representations from Transformers (BERT) models excel in Natural Language Processing (NLP) but are computationally expensive for large-scale biomedical applications.
• Existing biomedical BERT models like BioBERT and PubMedBERT offer advanced capabilities but retain significant parameter counts.
• There is a need for computationally efficient yet high-performing language models in the biomedical domain.

Approach:

• Developed Bioformer, a compact BERT variant specifically for biomedical text mining, reducing model size by 60% compared to BERT-Base.
• Pretrained two Bioformer models from scratch using a biomedical vocabulary on PubMed abstracts and full-text articles.
• Evaluated Bioformer against BioBERT and PubMedBERT on 15 benchmark datasets across four biomedical NLP tasks: named entity recognition, relation extraction, question answering, and document classification.

Key Points:

• Bioformer models achieve performance comparable to PubMedBERT, with only a 0.1% and 0.9% accuracy decrease, respectively.
• Both Bioformer variants significantly outperform BioBERT and demonstrate 2-3 fold speed improvements over PubMedBERT and BioBERT.
• Bioformer has been successfully deployed in PubTator Central for gene annotation across millions of biomedical articles.

Conclusions:

• Bioformer offers a computationally efficient and effective solution for biomedical NLP tasks, maintaining high accuracy with reduced model size.
• The model's speed and performance make it suitable for large-scale biomedical text mining and analysis.
• Public availability of Bioformer facilitates its adoption and further development in the research community.