Benchmarking for biomedical natural language processing tasks with a domain specific ALBERT.

Area of Science:

• Biomedical Natural Language Processing (BioNLP)
• Machine Learning
• Computational Linguistics

Background:

• The growing volume of biomedical text necessitates advanced Natural Language Processing (NLP) tools.
• Existing domain-specific language models (LMs) often use BERT architecture with limitations and unproven generalizability.
• A lack of baseline results hinders progress in common BioNLP tasks.

Purpose of the Study:

• To develop and evaluate BioALBERT, a domain-specific adaptation of the lite bidirectional encoder representations from transformers (ALBERT) model.
• To establish new state-of-the-art benchmarks for common BioNLP tasks.
• To provide a robust and generalizable model for the BioNLP community.

Main Methods:

• Trained 8 variants of BioALBERT on biomedical (PubMed, PubMed Central) and clinical (MIMIC-III) corpora.
• Fine-tuned BioALBERT variants on 6 different BioNLP tasks across 20 benchmark datasets.
• Evaluated model performance against existing state-of-the-art methods.

Main Results:

• A large BioALBERT variant trained on PubMed achieved state-of-the-art performance on 5 out of 6 BioNLP tasks.
• Significant improvements were observed in named-entity recognition (+11.09% BLURB score) and question answering (+2.83% BLURB score).
• Five BioALBERT variants outperformed previous models on 17 out of 20 benchmark datasets, indicating robustness and generalizability.

Conclusions:

• BioALBERT demonstrates superior performance and generalizability across a wide range of BioNLP tasks.
• The model establishes new state-of-the-art results, providing valuable baselines for future research.
• Freely available BioALBERT reduces computational burden and facilitates advancements in the BioNLP community.