Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.

Area of Science:

• Artificial intelligence
• Computational biology
• Bioinformatics

Background:

• Unsupervised learning and large datasets drive advances in AI representation learning.
• The life sciences are experiencing exponential growth in sequencing data, offering insights into natural sequence diversity.
• Protein language modeling, scaled with evolutionary data, is a key step for predictive and generative AI in biology.

Purpose of the Study:

• To train a deep contextual language model on a massive dataset of protein sequences.
• To explore the potential of unsupervised learning for biological representation learning.
• To develop AI tools for understanding and predicting protein properties and functions.

Main Methods:

• Trained a deep contextual language model using unsupervised learning.
• Utilized a dataset of 86 billion amino acids from 250 million protein sequences.
• Analyzed the learned representation space for encoded biological information.

Main Results:

• The model learned representations encoding biological properties directly from sequence data.
• The representation space exhibited multiscale organization, from amino acid properties to protein homology.
• Encoded information about secondary and tertiary protein structure was identifiable.
• Achieved state-of-the-art results in predicting mutational effects and secondary structure.
• Improved features for predicting long-range protein contacts.

Conclusions:

• Unsupervised learning on large-scale evolutionary sequence data enables powerful protein representation learning.
• Learned protein representations generalize across diverse biological prediction tasks.
• This approach advances predictive and generative AI capabilities for biological applications.