Understanding Language Model Scaling on Protein Fitness Prediction.

Area of Science:

• Computational Biology
• Machine Learning
• Protein Engineering

Background:

• Protein language models estimate sequence likelihoods (p(sequence)) for mutation effect prediction and protein design.
• Larger deep learning models are generally assumed to perform better across various tasks.

Purpose of the Study:

• To investigate the scalability of protein language models for fitness prediction.
• To understand how model size, training data, and stochasticity affect predicted p(sequence) and its relation to real protein fitness.

Main Methods:

• Analysis of protein language model performance on fitness prediction across different model sizes and training datasets.
• Evaluation of how predicted p(sequence) correlates with evolutionary patterns in homologous sequences.

Main Results:

• Model performance in fitness prediction decreases beyond a certain size, contrary to general deep learning trends.
• Model size, training data, and stochastic elements can bias predicted p(sequence) away from actual protein fitness.
• Optimal fitness prediction occurs when p(sequence) matches evolutionary patterns at a moderate level; extreme likelihoods lead to uniform predictions, failing to capture the fitness landscape.

Conclusions:

• Larger protein models tend to predict higher p(sequence), potentially exceeding the optimal moderate range and reducing fitness prediction accuracy.
• Findings clarify the scaling behavior of protein models for fitness prediction and offer practical guidelines for model development and application.