ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning

Area of Science:

• Computational biology
• Bioinformatics
• Natural Language Processing (NLP)
• Protein science

Background:

• Large datasets of protein sequences are valuable resources for developing predictive models.
• Natural Language Processing (NLP) models, specifically Language Models (LMs), show promise for analyzing biological data.
• Protein Language Models (pLMs) can potentially advance prediction tasks with lower computational costs.

Purpose of the Study:

• To train and evaluate various Language Models (LMs) on extensive protein sequence data.
• To investigate the capability of protein LMs (pLMs) to capture biophysical features from unlabeled protein sequences.
• To assess the performance of pLM-derived embeddings in downstream prediction tasks such as protein secondary structure and subcellular localization.

Main Methods:

• Trained auto-regressive (Transformer-XL, XLNet) and auto-encoder (BERT, Albert, Electra, T5) models on UniRef and BFD datasets (up to 393 billion amino acids).
• Utilized the Summit supercomputer with 5616 GPUs and TPU Pods (up to 1024 cores) for model training.
• Applied dimensionality reduction to analyze pLM-embeddings and validated their performance on secondary structure prediction, subcellular localization, and membrane/water-soluble classification.

Main Results:

• Dimensionality reduction showed that raw pLM-embeddings from unlabeled data captured biophysical features of protein sequences.
• pLM embeddings achieved high accuracy in predicting protein secondary structure (Q3=81%-87%) and subcellular location (Q10=81%, Q2=91%).
• The ProtT5 model, using pLM embeddings, outperformed state-of-the-art methods for secondary structure prediction without relying on multiple sequence alignments (MSAs) or evolutionary information.

Conclusions:

• Protein Language Models (pLMs) effectively learn underlying patterns ('grammar') in protein sequences.
• pLM embeddings serve as powerful, general-purpose features for diverse protein prediction tasks.
• These findings bypass the need for computationally expensive evolutionary data, opening new avenues in protein bioinformatics.