ProteinBERT: a universal deep-learning model of protein sequence and function.

Area of Science:

• Computational Biology
• Bioinformatics
• Artificial Intelligence in Biology

Background:

• Deep language models excel in natural language tasks but require adaptation for biological sequences.
• Existing models are optimized for text, not specifically for the complexities of protein data.

Purpose of the Study:

• Introduce ProteinBERT, a deep language model specifically engineered for protein sequence analysis.
• Develop an efficient and flexible framework for protein prediction tasks.

Main Methods:

• A novel pretraining scheme combining language modeling with Gene Ontology (GO) annotation prediction.
• Incorporated novel architectural elements for efficient processing of long protein sequences.
• Designed an architecture capturing both local and global protein sequence representations.

Main Results:

• ProteinBERT achieves near state-of-the-art and sometimes superior performance on diverse protein property benchmarks.
• Demonstrated effectiveness across protein structure, post-translational modifications, and biophysical attributes.
• Outperforms competing deep-learning methods with a significantly smaller and faster model.

Conclusions:

• ProteinBERT offers an efficient framework for rapid development of protein predictors.
• The model is effective even when trained on limited labeled protein data.
• Enables advancements in understanding and predicting diverse protein characteristics.