Scalable embedding fusion with protein language models: insights from benchmarking text-integrated representations

Area of Science:

• Computational biology
• Bioinformatics
• Machine learning in biology

Background:

• Protein language models (pLMs) utilize pretrained embeddings for transfer learning in biology.
• Standard pLM objectives may generate representations misaligned with downstream biological tasks.
• Increasing model size does not guarantee improved representation quality.

Purpose of the Study:

• To investigate strategies for enhancing protein language model representations.
• To improve the utility of pLMs for diverse biological applications.
• To address limitations of current pLM embedding strategies.

Main Methods:

• Integrating biological text annotations via contrastive learning to create text-integrated pLMs (tpLMs).
• Employing embedding fusion to combine representations from multiple pLMs.
• Developing and applying a greedy forward selection algorithm for efficient embedding subset identification.

Main Results:

• No single pLM or tpLM consistently outperformed others across all tested tasks.
• Fusion of tpLM embeddings enhanced performance on most biological tasks.
• The greedy forward selection algorithm efficiently identified near-optimal embedding subsets, achieving state-of-the-art results in homologous sequence recovery and protein-protein interaction prediction.

Conclusions:

• Embedding fusion is a practical and scalable method for improving protein representations.
• Text integration and embedding fusion offer complementary strategies for advancing pLM capabilities.
• The developed greedy algorithm overcomes computational bottlenecks associated with embedding fusion.