Evaluating Protein Transfer Learning with TAPE.

Area of Science:

• Computational biology
• Machine learning
• Bioinformatics

Background:

• Machine learning for protein sequences is growing, but lacks standardized datasets and evaluation for semi-supervised learning.
• High costs of supervised labels necessitate effective semi-supervised approaches for protein analysis.

Purpose of the Study:

• Introduce Tasks Assessing Protein Embeddings (TAPE), a benchmark for semi-supervised protein representation learning.
• Provide standardized tasks and data splits for biologically relevant generalization.
• Benchmark current semi-supervised methods and identify areas for improvement.

Main Methods:

• Curated five biologically relevant semi-supervised learning tasks across protein biology domains.
• Developed specific training, validation, and test splits for robust evaluation.
• Benchmarked various semi-supervised protein representation learning approaches, including recent and canonical techniques.

Main Results:

• Self-supervised pretraining significantly improved performance across most models and tasks, sometimes doubling results.
• Despite improvements, self-supervised features occasionally underperformed state-of-the-art non-neural methods.
• Identified a performance gap indicating potential for novel architectures and modeling paradigms.

Conclusions:

• TAPE provides a standardized framework to advance semi-supervised protein learning.
• Self-supervised pretraining is beneficial but not a complete solution for protein representation.
• Further research is needed in machine learning architectures to fully capture biological sequence signals.