PTM-Mamba: A PTM-Aware Protein Language Model with Bidirectional Gated Mamba Blocks

Area of Science:

• * Computational biology and bioinformatics.
• * Protein structure and function analysis.
• * Machine learning in genomics and proteomics.

Background:

• * Proteins perform essential cellular functions, with post-translational modifications (PTMs) significantly impacting their diversity and regulation.
• * Existing protein language models (pLMs) like ESM-2 and ProtT5 lack the ability to process or account for PTMs, limiting their scope.
• * Proteomic diversity is vast, and current pLMs trained on limited sequence data do not capture the full spectrum of protein variations, especially those arising from PTMs.

Purpose of the Study:

• * To develop the first protein language model capable of uniquely inputting and representing post-translationally modified (PTM) protein sequences.
• * To enhance the modeling of protein sequences by integrating PTM information, addressing a critical gap in current pLMs.
• * To create a PTM-aware model that facilitates downstream applications in protein design and analysis.

Main Methods:

• * Integration of a comprehensive set of post-translational modification (PTM) tokens into a protein language model vocabulary.
• * Leveraging structured state space models (SSMs), specifically Mamba, for efficient sequence modeling.
• * Fusion of bidirectional Mamba blocks with ESM-2 embeddings using a novel gating mechanism to create the PTM-Mamba model.

Main Results:

• * The developed PTM-aware pLM, PTM-Mamba, demonstrates improved performance on PTM-specific tasks compared to the state-of-the-art ESM-2.
• * PTM-Mamba successfully inputs and represents both wild-type and PTM-containing protein sequences.
• * The model's ability to uniquely handle PTMs opens new avenues for downstream modeling and design of modified proteins.

Conclusions:

• * PTM-Mamba represents a significant advancement in protein language modeling by uniquely incorporating post-translational modifications.
• * The model's PTM-awareness enhances the accurate representation of protein sequences, crucial for understanding biological processes.
• * PTM-Mamba is poised to drive innovation in computational biology, enabling novel applications for modified proteins.