Extending Protein Language Models to a Viral Genomic Scale Using Biologically Induced Sparse Attention

Area of Science:

• Computational biology
• Deep learning
• Genomics

Background:

• Transformer models have advanced protein sequence analysis but typically ignore genomic context.
• Existing protein language models trained on individual proteins miss crucial inter-protein dependencies within genomes.
• Protein-protein interactions are known to span genomic regions, highlighting limitations of single-protein analysis.

Purpose of the Study:

• To develop a novel deep learning approach for protein sequence analysis that incorporates genomic context.
• To extend transformer models to process entire viral genomes, capturing long-range dependencies.
• To improve protein function and interaction prediction by learning from genome-wide sequence information.

Main Methods:

• Developed a long-context protein language model trained on complete viral genomes.
• Employed a biologically informed sparse attention mechanism to infer inter-protein links.
• Utilized a semi-supervised approach supporting sequences up to 61,000 amino acids.

Main Results:

• The genome-wide model demonstrated improved prediction of masked amino acids compared to single-protein models.
• Downstream task performance, including protein function prediction, was enhanced.
• Inferred inter-protein links showed correlation with established interaction databases.

Conclusions:

• Training protein language models on entire viral genomes effectively captures inter-protein relationships and long-range dependencies.
• The proposed long-context model offers significant improvements over traditional single-protein approaches for genomic sequence analysis.
• This genome-wide learning strategy holds promise for advancing biological understanding and prediction tasks in virology and beyond.