Scalable embedding fusion with protein language models: insights from benchmarking text-integrated representations.

Area of Science:

• Computational biology
• Bioinformatics
• Machine learning in biology

Background:

• Protein language models (pLMs) utilize pretrained embeddings for various biological applications.
• Standard training objectives may yield suboptimal representations for downstream tasks.
• Model scaling does not always guarantee improved representation quality.

Purpose of the Study:

• To investigate strategies for enhancing protein language model representations.
• To improve the utility of pLMs in data-scarce biological settings.
• To develop efficient methods for combining multiple pLM embeddings.

Main Methods:

• Integrating biological text annotations with pLMs using contrastive learning.
• Developing an embedding fusion strategy to combine representations from multiple pLMs.
• Proposing a greedy forward selection algorithm for efficient embedding subset identification.

Main Results:

• Text-integrated pLMs (tpLMs) and large-scale pLMs were benchmarked across six diverse biological tasks.
• No single model consistently outperformed others across all tasks.
• Embedding fusion improved performance on most tasks, with greedy forward selection efficiently identifying near-optimal subsets.
• New state-of-the-art results were achieved in homologous sequence recovery and protein-protein interaction prediction.

Conclusions:

• Embedding fusion is a practical and scalable strategy for improving protein representations.
• Integrating text data and combining embeddings offers significant advantages for pLM performance.
• Efficient algorithms are crucial for leveraging the benefits of embedding fusion in large-scale applications.