Medium-sized protein language models perform well at transfer learning on realistic datasets.

Area of Science:

• Computational biology
• Bioinformatics
• Machine learning in protein science

Background:

• Protein language models (pLMs) provide insights into protein evolution and structure.
• Large pLMs like ESM-2 (15B parameters) capture complex patterns but are computationally intensive.
• Model size impacts performance in transfer learning applications.

Purpose of the Study:

• To systematically evaluate the effect of protein language model size on transfer learning performance.
• To compare different embedding compression methods for feature extraction.
• To identify optimal pLM configurations balancing performance and computational cost.

Main Methods:

• Systematic evaluation of various ESM-style protein language models across diverse biological datasets.
• Transfer learning using feature extraction from pre-trained pLMs.
• Comparison of embedding compression techniques, including mean embeddings.

Main Results:

• Larger protein language models do not consistently outperform smaller ones, especially with limited data.
• Medium-sized models (ESM-2 650M, ESM C 600M) showed strong performance, slightly below larger models.
• Mean embeddings outperformed other compression methods for transfer learning.

Conclusions:

• ESM C 600M combined with mean embeddings presents an optimal trade-off between performance and efficiency.
• This combination is a practical and scalable solution for transfer learning in biological applications.
• Model size is not the sole determinant of success in pLM-based transfer learning.