Intrinsic dataset features drive mutational effect prediction by protein language models.

Area of Science:

• Computational biology
• Protein engineering
• Machine learning in biology

Background:

• Protein language models (pLMs) are increasingly used for predicting protein fitness landscapes.
• Performance of pLMs varies significantly across different datasets, with reasons often unclear.

Purpose of the Study:

• To investigate the factors influencing the performance of pLMs in predicting protein fitness landscapes.
• To compare pLM performance on viral versus cellular deep-mutational-scanning (DMS) datasets.
• To identify intrinsic dataset features that impact predictive accuracy.

Main Methods:

• Evaluated supervised transfer learning using multiple pLMs on 41 viral and 33 cellular DMS datasets.
• Analyzed dataset composition using metrics like relative variability of site means (RVSM) and fraction of highly variable sites (FHVS).
• Compared pLM performance against a baseline model predicting site mean fitness and assessed data leakage by splitting training/test data by site.

Main Results:

• Consistently lower predictive performance was observed on viral datasets compared to cellular datasets, irrespective of model or transfer learning strategy.
• A simple baseline model predicting site mean fitness often matched or outperformed supervised pLMs.
• Dataset variability metrics (RVSM, FHVS) explained differences between viral and cellular datasets and constrained model performance.
• Supervised models frequently relied on site-specific effects rather than general mutational patterns, especially when data leakage was controlled.

Conclusions:

• Dataset composition, particularly site variability, is a primary driver of pLM predictive success for mutational effects, not model architecture or training strategy.
• Current pLMs have limitations in capturing critical site mutational constraints for fitness prediction.
• Existing benchmarks may overestimate pLM performance due to data leakage and failure to account for site effects.