Trainable subnetworks reveal insights into structure knowledge organization in protein language models

Area of Science:

• Computational biology
• Structural bioinformatics
• Machine learning in biology

Background:

• Protein language models (PLMs) excel at structure-related tasks.
• The extent to which PLMs factorize protein structural information is not well understood.

Purpose of the Study:

• To investigate the factorization of protein structural categories within PLM parameters.
• To assess the influence of structural factorization on protein structure prediction.

Main Methods:

• Trained 39 PLM subnetworks by masking weights related to specific structural categories.
• Utilized CATH taxonomy and secondary structure elements for annotation.
• Assessed downstream structure prediction performance using factorized PLM representations.

Main Results:

• PLMs show high sensitivity to sequence-level features.
• Models can disentangle both coarse and fine-grained structural information.
• Structure prediction performance is highly responsive to factorized representations; minor LM performance changes significantly impact prediction.

Conclusions:

• PLMs exhibit significant structural factorization capabilities.
• Understanding feature entanglement is crucial for improving PLM alignment with biological concepts.
• This framework aids in studying and enhancing PLMs for biological applications.