Boost Protein Language Model with Injected Structure Information Through Parameter Efficient Fine-tuning

Area of Science:

• Computational biology
• Bioinformatics
• Structural biology

Background:

• Large-scale Protein Language Models (PLMs) like ESM have advanced protein understanding but often lack structural information.
• Current PLMs primarily use residue sequences, limiting their ability to capture complex protein structures and functions.
• Integrating structural data into PLMs offers a significant opportunity to enhance their predictive power in biomedical applications.

Purpose of the Study:

• To introduce Structure Information Injecting Tuning (SI-Tuning), a novel parameter-efficient fine-tuning method for PLMs.
• To effectively integrate structural information into existing PLMs without altering their core parameters.
• To improve the performance of PLMs on various downstream tasks by leveraging structural features.

Main Methods:

• SI-Tuning is a parameter-efficient fine-tuning approach that keeps pre-trained PLM parameters frozen.
• Task-specific vectors are optimized for input embedding and attention maps.
• Structural features, such as dihedral angles and distance maps, are used to derive these vectors for injecting structural information.

Main Results:

• SI-Tuning demonstrated significant performance improvements across multiple downstream tasks on the 650M ESM-2 model.
• Achieved 93.95% accuracy on DeepLoc binary classification and 76.05% on Metal Ion Binding.
• Outperformed SaProt, a PLM with integrated structural modeling, highlighting the effectiveness of SI-Tuning.

Conclusions:

• SI-Tuning is an effective method for enhancing PLM performance by incorporating structural information in a parameter-efficient manner.
• The approach offers substantial computational efficiency, making it valuable for broad biomedical applications.
• SI-Tuning represents a significant advancement in leveraging structural data for improved protein language model utility.