Hyena architecture enables fast and efficient protein language modeling.

Area of Science:

• Computational biology
• Bioinformatics
• Machine learning

Background:

• Deep learning models, particularly Transformers, have advanced biological sequence analysis.
• Transformer models face computational bottlenecks due to the attention mechanism's quadratic complexity (O(L^2)).
• High computational costs limit the efficiency of current models for large biological datasets.

Purpose of the Study:

• To introduce ProtHyena, an efficient protein language model.
• To address the computational limitations of Transformer-based models in protein sequence analysis.
• To develop a model with subquadratic complexity for faster and more memory-efficient protein modeling.

Main Methods:

• Utilized the Hyena operator, alternating subquadratic long convolutions with element-wise gating.
• Implemented an architecture that circumvents attention mechanisms.
• Developed a model with only 1.6 million parameters.

Main Results:

• Achieved state-of-the-art and comparable performance across 8 diverse downstream tasks.
• Demonstrated significant reductions in computational complexity to subquadratic levels.
• Enabled faster and more memory-efficient protein sequence modeling.

Conclusions:

• ProtHyena offers a highly efficient solution for protein language modeling.
• The Hyena operator effectively overcomes Transformer limitations in biological sequence analysis.
• ProtHyena presents a promising approach for rapid and efficient protein sequence analysis.