A systematic assessment of single-cell language model configurations

Transformers pre-trained on single-cell transcriptomic data have recently been applied to a series of tasks, earning them the title of foundation models. However, recent benchmarks indicate that current iterations of single-cell foundation models may often be surpassed by simpler task-specific models. As all currently published models in this class employ vastly different strategies, it is impossible to determine which practices drive their success (or failure). To help steer research in a more productive direction, we present a framework for the study of single-cell foundation models: bento-sc (BENchmarking Transformer-Obtained Single-Cell representations). We use bento-sc to perform a large-scale benchmarking of single-cell language model (scLM) configurations. By isolating parts of the pre-training scheme one by one, we define best practices for scLM construction. While comparisons with baselines indicate that scLMs do not yet offer the generational leap in prediction performances promised by many foundation models, we identify key design choices leading to improved performance. Namely, the best scLMs are obtained by: (i) minimally processing counts for input, (ii) using reconstruction losses that exploit known count distributions, (iii) masking (up to high rates), and (iv) combining different pre-training tasks/losses. All code supporting this study is distributed on PyPI and is packaged under: https://github.com/gdewael/bento-sc.