Tokenized and continuous embedding compressions of protein sequence and structure.

Area of Science:

• Computational Biology
• Machine Learning
• Structural Bioinformatics

Background:

• Current protein machine learning models often focus on sequence or structure independently.
• Integrating both sequence and structure information is crucial for comprehensive protein representation.

Purpose of the Study:

• To develop a compressed, interpretable, and flexible representation of joint protein sequence and structure distributions.
• To introduce CHEAP embeddings derived from the ESMFold model's latent space.

Main Methods:

• Compressing the latent space of the ESMFold protein folding model using continuous and discrete schemes.
• Evaluating the retention of structural information and performance on downstream tasks like function and localization prediction.

Main Results:

• Continuous compression reduced ESMFold latent space by 128x (channel) and 8x (length) while preserving <2 Å accuracy.
• Discrete compression created a tokenized all-atom structure vocabulary with high reconstruction accuracy.
• CHEAP embeddings performed competitively on protein function and localization benchmarks.

Conclusions:

• CHEAP embeddings offer mechanistic interpretability and a flexible representation of protein sequence and structure.
• This approach democratizes access to large model representations.
• CHEAP enables diverse downstream applications including protein generation, search, and prediction.