Protein language models learn evolutionary statistics of interacting sequence motifs.

Area of Science:

• Computational biology
• Structural bioinformatics
• Machine learning in biology

Background:

• Protein language models (pLMs) are powerful tools for predicting protein structure and function.
• The extent to which pLMs understand protein biophysics remains an open question.
• pLM-based predictors have shown limitations, erroneously predicting nonphysical structures for protein isoforms.

Purpose of the Study:

• Investigate the sequence context required for contact predictions in the pLM Evolutionary Scale Modeling (ESM-2).
• Understand how ESM-2 stores information for predicting residue-residue contacts.
• Clarify the underlying mechanisms and limitations of pLMs in structural prediction.

Main Methods:

• Employed a "categorical Jacobian" calculation to analyze ESM-2's internal representations.
• Compared different sequence masking strategies to assess information storage.
• Evaluated the model's ability to recover predicted contacts based on sequence context.

Main Results:

• ESM-2 stores statistics of coevolving residues, analogous to Markov Random Fields and Multivariate Gaussian models.
• Local windows of sequence information are most effective for ESM-2 to predict contacts.
• The model appears to predict contacts by storing pairwise contact motifs.

Conclusions:

• Current pLMs like ESM-2 rely on storing coevolutionary statistics and local motifs for contact prediction.
• These findings highlight limitations in the fundamental biophysical understanding of pLMs.
• Further research into pLM mechanisms is crucial for advancing protein structure prediction and design.