MULAN: multimodal protein language model for sequence and structure encoding.

Area of Science:

• Computational biology
• Bioinformatics
• Artificial intelligence in biology

Background:

• Protein language models (PLMs) primarily utilize protein sequences for high-quality representations.
• Incorporating protein structural information is crucial for various prediction tasks, driving interest in structure-aware PLMs.
• Existing structure-aware PLMs often require training from scratch or substantial parameter additions for structure encoding.

Purpose of the Study:

• To introduce MULAN, a multimodal protein language model (PLM) that integrates both sequence and angle-based structural information.
• To develop a parameter-efficient approach for incorporating structural awareness into pre-trained PLMs.
• To evaluate MULAN's performance across diverse downstream tasks compared to existing models.

Main Methods:

• Developed MULAN, a multimodal PLM featuring a pre-trained sequence encoder and a parameter-efficient Structure Adapter.
• Fused the sequence encoder and Structure Adapter, training them together for multimodal representation learning.
• Evaluated MULAN on nine downstream prediction tasks, assessing its performance against sequence-only and structure-aware baselines.

Main Results:

• MULAN models demonstrated improved performance across various sizes compared to sequence-only ESM2 and structure-aware SaProt.
• Significant improvements were observed in protein-protein interaction prediction, with AUROC increasing by up to 0.12.
• MULAN achieved enhanced structural awareness through efficient fine-tuning of existing PLMs, avoiding costly training from scratch.

Conclusions:

• MULAN offers an effective and computationally inexpensive method to imbue protein representations with structural awareness.
• The proposed Structure Adapter provides a parameter-efficient way to integrate structural information into PLMs.
• MULAN represents a valuable advancement for structure-aware protein modeling and prediction tasks.