Understanding language model scaling for protein fitness prediction

Area of Science:

• Computational biology
• Deep learning for proteins
• Bioinformatics

Background:

• Protein language models (PLMs) estimate sequence likelihood (p(sequence)) for fitness prediction.
• Larger deep learning models are generally assumed to perform better.
• PLM performance in fitness prediction declines beyond a certain model size.

Purpose of the Study:

• Investigate the impact of model size on protein fitness prediction.
• Clarify the scalability of PLMs for predicting mutation effects.
• Provide guidelines for optimizing PLM application in protein design.

Main Methods:

• Analyzed the relationship between PLM size and predicted sequence likelihood (p(sequence)).
• Evaluated how model size, training data, and stochasticity affect fitness landscape representation.
• Compared predicted p(sequence) with evolutionary patterns in homologous proteins.

Main Results:

• Model size, training data, and stochasticity can bias predicted p(sequence) from true protein fitness.
• Optimal fitness prediction occurs when predicted p(sequence) moderately matches evolutionary patterns.
• Larger models tend to predict excessively high p(sequence), reducing fitness prediction accuracy.
• Extreme predicted likelihoods lead to uniform predictions for mutations, failing to capture the real fitness landscape.

Conclusions:

• Protein language model performance in fitness prediction is not solely dependent on size.
• Moderate predicted sequence likelihoods are crucial for accurately reflecting protein fitness landscapes.
• Findings offer practical guidance for developing and applying protein models in computational biology and protein engineering.