Masked Language Modeling for Resource Constrained Biological Natural Language Processing

Area of Science:

• Bioinformatics
• Computational Biology
• Natural Language Processing

Background:

• Recent Natural Language Processing (NLP) advances, particularly deep contextualized embedders, have surpassed traditional word vector models.
• Protein sequences, composed of 20 amino acids, can be viewed as a natural language, crucial for understanding biological functions.
• Effective vector representation and model architecture are vital for biological tasks.

Purpose of the Study:

• To develop an efficient protein sequence embedding method using unlabelled data.
• To evaluate the performance of a novel Convolution and Gated Recurrent Network (CGRN) embedder trained with Masked Language Modeling (MLM).
• To analyze the impact of different model architectures on downstream biological tasks.

Main Methods:

• Utilized unlabelled protein sequences for training.
• Employed a Convolution and Gated Recurrent Network (CGRN) architecture.
• Applied the Masked Language Modeling (MLM) technique for embedder training.

Main Results:

• Achieved a 73.1% F1-score (Q8) on Secondary Structure Prediction (SSP).
• Attained an 84% F1-score on Intrinsically Disordered Region Prediction (IDRP).
• Demonstrated significant performance boosts under resource-constrained settings.

Conclusions:

• The CGRN embedder trained with MLM shows strong performance on downstream tasks like SSP and IDRP.
• The choice of model architecture significantly impacts performance based on the specific biological task.
• This approach offers an effective solution for protein sequence analysis, especially in resource-limited environments.