SPRoBERTa: protein embedding learning with local fragment modeling.

Area of Science:

• Computational biology
• Bioinformatics
• Structural biology

Background:

• Understanding protein structure and function is crucial for human biology.
• Limited annotated protein data necessitates advanced computational methods.
• Current self-supervised methods often overlook local semantic patterns in protein sequences.

Purpose of the Study:

• To develop a novel pre-training approach, SPRoBERTa, for enhanced protein embedding learning.
• To address limitations in representing protein sequences by incorporating local fragment patterns.
• To create a versatile framework for various protein-related prediction tasks.

Main Methods:

• Developed an unsupervised protein tokenizer to capture local fragment patterns.
• Introduced a deep pre-training framework for learning protein embeddings.
• Fine-tuned the pre-trained model for amino acid-level, amino acid pair-level, and protein-level prediction tasks.

Main Results:

• SPRoBERTa achieved significant improvements across diverse protein prediction tasks.
• The proposed method outperformed existing state-of-the-art approaches.
• Ablation studies validated the effectiveness of the protein tokenizer and training framework.

Conclusions:

• SPRoBERTa offers a powerful new approach for protein representation learning.
• Incorporating local sequence semantics enhances protein embedding quality.
• The method demonstrates broad applicability and superior performance in computational biology.