Convolutions are competitive with transformers for protein sequence pretraining.

Area of Science:

• Bioinformatics
• Computational Biology
• Machine Learning

Background:

• Pretrained protein sequence models enhance bioinformatics tasks but often use transformer architectures.
• Transformer models have quadratic scaling limitations, restricting protein sequence length.
• Current state-of-the-art models face sequence length constraints.

Purpose of the Study:

• To investigate if convolutional neural network (CNN) architectures can match transformer effectiveness in protein language models.
• To explore CNNs' linear scaling for handling longer protein sequences.
• To assess CNNs' performance in protein sequence modeling.

Main Methods:

• Utilized masked language model pretraining for CNN architectures.
• Compared CNN performance against transformer models on downstream prediction tasks.
• Evaluated models on their ability to process sequences exceeding current transformer limits.

Main Results:

• CNNs demonstrated competitive, and sometimes superior, performance compared to transformers.
• CNNs maintained strong performance on protein sequences longer than transformer models can handle.
• The study confirmed CNNs' linear scaling with sequence length.

Conclusions:

• CNN architectures are a viable and efficient alternative to transformers for protein language modeling.
• Computational efficiency in protein modeling can be enhanced using CNNs without performance loss.
• Disentangling pretraining tasks from model architecture is crucial for advancing protein language models.