ProGen2: Exploring the boundaries of protein language models

Area of Science:

• Computational Biology
• Artificial Intelligence in Biology
• Protein Engineering

Background:

• Attention-based models excel in protein sequence classification and generation for AI-driven protein design.
• Understanding the impact of large-scale models and data on protein model development is crucial.

Purpose of the Study:

• Introduce ProGen2, a suite of large-scale protein language models.
• Investigate the role of model size and diverse sequence data in protein modeling.
• Establish state-of-the-art performance in protein sequence tasks.

Main Methods:

• Developed ProGen2 models up to 6.4B parameters.
• Trained models on diverse datasets exceeding one billion protein sequences (genomic, metagenomic, immune repertoire).
• Evaluated performance on sequence distribution, novel sequence generation, and protein fitness prediction.

Main Results:

• ProGen2 models achieved state-of-the-art performance across key tasks.
• Demonstrated superior ability in capturing evolutionary sequence distributions.
• Successfully generated novel, viable protein sequences and predicted protein fitness without fine-tuning.

Conclusions:

• Large model sizes and extensive protein sequence data are key for effective protein modeling.
• Data distribution is a critical factor for protein sequence model performance.
• Open-sourced ProGen2 models and code facilitate widespread adoption in protein engineering.