Heimdall: A Modular Framework for Tokenization in Single-Cell Foundation Models.

Area of Science:

• Computational Biology
• Genomics
• Machine Learning

Background:

• Foundation models are increasingly used for single-cell RNA sequencing (scRNA-seq) data analysis.
• The performance of these models heavily relies on cell tokenization strategies, which are not well understood.
• Developing effective tokenization methods is critical for advancing scRNA-seq analysis.

Purpose of the Study:

• To introduce Heimdall, a framework and toolkit for evaluating tokenization strategies in single-cell foundation models (scFMs).
• To systematically assess the impact of different tokenization components on model performance.
• To provide a foundation for reproducible research and development of next-generation scFMs.

Main Methods:

• Decomposition of scFMs into modular components: gene identity encoder (F_G), expression encoder (F_E), and cell sentence constructor (F_C).
• Evaluation of tokenization strategies using a transformer model trained from scratch.
• Assessment across challenging transfer learning scenarios: cross-tissue, cross-species, and spatial gene-panel shifts, and reverse perturbation prediction.

Main Results:

• Tokenization choices have minimal impact on in-distribution data but are critical under distribution shifts.
• Gene identity (F_G) and token order significantly improve generalization.
• Expression encoder (F_E) provides additional performance gains.
• Recombining existing strategies can enhance model generalization.

Conclusions:

• Heimdall provides a standardized approach for evaluating single-cell tokenization strategies.
• Tokenization design is a key determinant of scFM performance, particularly in out-of-distribution settings.
• Heimdall accelerates the development of more robust and generalizable scFMs for single-cell analysis.