Extending Protein Language Models to a Viral Genomic Scale Using Biologically Induced Sparse Attention

Area of Science:

• Genomics
• Deep Learning
• Bioinformatics

Background:

• Transformer models have advanced protein sequence analysis.
• Current protein language models typically analyze individual proteins, neglecting genomic context.
• Protein-protein interactions are crucial and span genomic regions.

Purpose of the Study:

• To develop a novel protein language model with extended context size across entire viral genomes.
• To capture long-range interprotein interactions and integrate information from distant proteins within a genome.
• To improve protein sequence analysis by considering genomic context.

Main Methods:

• Proposed a novel approach extending transformer models' context size to entire viral genomes.
• Employed a sparse attention mechanism based on protein-protein interactions.
• Utilized a semi-supervised approach for training on sequences up to 61,000 amino acids.

Main Results:

• The genome-wide protein language model captures long-range interprotein interactions.
• Generated embeddings significantly surpass those from single-protein models.
• Outperformed alternative large-context architectures and non-transformer frameworks.

Conclusions:

• Training protein language models on entire viral genomes is effective for capturing interprotein dependencies.
• The proposed long-context model offers substantial benefits for various protein analysis tasks.
• This genome-wide learning approach addresses limitations of single-protein analysis.