Improved biomedical word embeddings in the transformer era

Area of Science:

• Biomedical Natural Language Processing
• Computational Linguistics
• Bioinformatics

Background:

• Neural networks and word embeddings are central to modern Natural Language Processing (NLP).
• Pre-trained embeddings, like skip-gram and GLoVE, capture word and concept distributions from text corpora.
• These embeddings are crucial for various downstream NLP tasks, often fine-tuned with specific neural architectures.

Purpose of the Study:

• To develop improved biomedical word embeddings for enhanced downstream applications.
• To make these advanced biomedical embeddings publicly available.
• To address limitations in existing static word embeddings for biomedical research.

Main Methods:

• Jointly learned word and concept embeddings using the skip-gram method.
• Fine-tuned embeddings with correlational information from co-occurring Medical Subject Heading (MeSH) concepts in biomedical citations.
• Employed a transformer-based BERT architecture with a classification objective for MeSH pair co-occurrence.

Main Results:

• Demonstrated improved biomedical embeddings through both qualitative and quantitative evaluations.
• Achieved clear performance improvements across multiple datasets for word relatedness.
• Provided a more exhaustive evaluation of biomedical embeddings compared to previous efforts.

Conclusions:

• Repurposed a transformer architecture to enhance static biomedical word embeddings using concept correlations.
• Successfully improved static biomedical word embeddings.
• Released code and embeddings for public use to support downstream applications and research.