Intrinsic dataset features drive mutational effect prediction by protein language models

Protein language models (pLMs) are commonly used for predicting protein fitness landscapes, but their wide range of performance across datasets remains poorly understood. We evaluated supervised transfer learning on 41 viral and 33 cellular deep-mutational-scanning (DMS) datasets using embeddings from multiple pLMs. We observed consistently lower predictive performance on viral datasets compared to cellular datasets, independent of model architecture or transfer learning strategy. Surprisingly, a simple baseline model that predicted site mean fitness matched or outperformed supervised models on many datasets, highlighting the dominant role of site effects. Analysis of site variability using two metrics, relative variability of site means (RVSM) and fraction of highly variable sites (FHVS), revealed that patterns of fitness variation within and among sites constrain model performance and largely explain the observed differences between viral and cellular datasets. Moreover, splitting training and test data by site, rather than pooling, revealed that supervised models often rely on site effects rather than capturing broader mutational patterns. These findings highlight limitations of current pLMs for mutational effect prediction and suggest that dataset composition, rather than model architecture or training, is the primary driver of predictive success.