The cell as a token: high-dimensional geometry in language models and cell embeddings.

Area of Science:

• Computational Biology
• Bioinformatics
• Genomics

Background:

• Single-cell sequencing generates high-dimensional data representing cellular activity.
• Virtual cell models enrich cell representations using pre-trained patterns from cell atlases.
• Natural language processing (NLP) and single-cell analysis share data processing paradigms.

Purpose of the Study:

• To explore how NLP embedding structures can inform single-cell data analysis.
• To highlight the application of NLP foundation models to cell atlases and virtual cell models.
• To bridge advancements in NLP with the development of robust cell models.

Main Methods:

• Reviewing literature at the intersection of NLP and single-cell biology.
• Analyzing parallels in data tokenization and high-dimensional embedding spaces.
• Examining the influence of token context and low-dimensional manifolds on data interpretation.

Main Results:

• Both NLP and single-cell data involve partitioning information into tokens within vector spaces.
• Token context significantly impacts embedding space geometry in both fields.
• Low-dimensional manifolds are crucial for the robustness and interpretability of embedding spaces.
• NLP foundation model developments, like interpretability probes and in-context reasoning, offer valuable insights.

Conclusions:

• NLP techniques, particularly foundation models, can enhance the construction of cell atlases.
• Virtual cell models can benefit from NLP-inspired methods for improved representation learning.
• Cross-disciplinary insights from NLP can advance single-cell data analysis and interpretation.