Continual knowledge infusion into pre-trained biomedical language models.

Area of Science:

• Biomedical Natural Language Processing (bioNLP)
• Artificial Intelligence
• Computational Biology

Background:

• Biomedical language models (e.g., BioBERT, BioELMo) excel at concept representation for bioNLP tasks.
• These models struggle with rare words due to limited contextual information.
• Integrating external knowledge bases (KBs) can improve representation quality for sparse data.

Purpose of the Study:

• To develop a novel representation learning approach for biomedical language models.
• To effectively fuse semantic information from multiple biomedical knowledge bases.
• To address the challenge of learning high-quality representations for concepts with low context.

Main Methods:

• Proposed a progressive fusion strategy to integrate KB semantic information into pretrained biomedical language models.
• Developed a knowledge modeling strategy to encode hierarchical KB topological properties at a granular level.
• Implemented a continual learning technique for efficient concept representation updates, preserving model efficiency.

Main Results:

• The proposed approach generates knowledge-powered embeddings with high fidelity and learning efficiency.
• Experimental validation on bioNLP tasks demonstrated the approach's efficacy.
• The method successfully generated robust concept representations, particularly for rare biomedical terms.

Conclusions:

• The novel approach effectively enhances biomedical language models by leveraging knowledge bases.
• This method improves the learning of representations for concepts with limited contextual information.
• The findings suggest a promising direction for advancing bioNLP applications through knowledge-infused language models.