Simple, Efficient, and Scalable Structure-Aware Adapter Boosts Protein Language Models.

Area of Science:

• Computational Biology
• Bioinformatics
• Machine Learning

Background:

• Fine-tuning pretrained protein language models (PLMs) is crucial for downstream prediction tasks.
• Parameter-efficient fine-tuning techniques from natural language processing show promise but face challenges in life sciences.
• Adapting these techniques requires addressing differences in training strategies and data formats.

Purpose of the Study:

• To introduce SES-Adapter, a novel method for enhancing PLM representation learning in life sciences.
• To create structure-aware protein representations by combining PLM embeddings with structural sequence embeddings.
• To demonstrate the efficiency and scalability of SES-Adapter across various PLM architectures and tasks.

Main Methods:

• Developed SES-Adapter, a parameter-efficient fine-tuning method.
• Integrated protein language model embeddings with structural sequence embeddings.
• Evaluated SES-Adapter on diverse downstream tasks using 9 benchmark datasets and 2 types of folding structures.

Main Results:

• SES-Adapter improved downstream task performance by an average of 3% and up to 11% compared to vanilla PLMs.
• Achieved significant acceleration in convergence speed, averaging 362% and up to 1034%.
• Demonstrated improved training efficiency (approximately 2x) and robustness with low-quality predicted structures.

Conclusions:

• SES-Adapter is a compatible and effective method for enhancing PLMs across different architectures and tasks.
• The approach significantly boosts performance, accelerates convergence, and improves training efficiency.
• SES-Adapter shows promise for advancing protein representation learning, even with imperfect structural data.