Annotation Vocabulary (Might Be) All You Need.

Area of Science:

• Computational biology
• Bioinformatics
• Machine learning

Background:

• Protein Language Models (pLMs) are crucial for computational protein analysis, but their embeddings often focus on structural features.
• There is a need to incorporate broader biochemical properties into protein representations.

Purpose of the Study:

• To develop a novel method for creating protein embeddings that capture diverse biochemical properties.
• To train transformer models using an "Annotation Vocabulary" for enhanced protein representation and generation.
• To demonstrate the efficiency and effectiveness of this new approach in downstream tasks.

Main Methods:

• Engineered the "Annotation Vocabulary," a transformer-readable language of protein properties based on structured ontologies.
• Trained "Annotation Transformers" (AT) to predict masked protein properties solely from descriptions, independent of amino acid sequences.
• Developed a novel loss function and utilized efficient computation for training representation models.
• Proposed a new sequence alignment-based score for evaluating de novo generated protein sequences.
• Utilized AT representations in various model architectures for protein representation and generation.

Main Results:

• The premier representation model, CAMP, achieved state-of-the-art embeddings on 5 out of 15 datasets with competitive performance on others, demonstrating computational efficiency.
• The generative model, GSM, produced high alignment scores from annotation-only prompts, with many generated sequences showing significant BLAST hits and matching annotation properties.
• Annotation Vocabulary integration enhanced transformer models, offering a new numerical feature space based on protein descriptions.

Conclusions:

• The Annotation Vocabulary provides a powerful and efficient method for enhancing protein representation and generation using transformer models.
• This approach offers a promising pathway to replace traditional tokens with ontologies and knowledge graphs for specialized domain applications.
• The Annotation Vocabulary facilitates concise, accurate, and efficient protein descriptions, enabling novel approaches to protein annotation and design.