RegFormer: a single-cell foundation model powered by gene regulatory hierarchies

Area of Science:

• Computational biology
• Genomics
• Bioinformatics

Background:

• Single-cell RNA sequencing (scRNA-seq) offers high-resolution cellular diversity insights.
• Existing computational models struggle with regulatory priors, data sparsity, and long gene sequences.

Purpose of the Study:

• To introduce RegFormer, a novel foundation model for single-cell data analysis.
• To overcome limitations of Transformer architectures in scalability and context length using Mamba-based state-space modeling and gene regulatory networks (GRNs).

Main Methods:

• RegFormer utilizes dual gene embeddings (value and token) within a GRN-guided order.
• The model is pretrained on a large dataset of 25 million human single cells across 45 tissues.
• It combines GRNs with Mamba-based state-space modeling for efficient long-sequence processing.

Main Results:

• RegFormer demonstrates superior scalability and biological fidelity compared to scGPT, Geneformer, scFoundation, and scBERT.
• Achieved higher clustering accuracy, improved batch integration, and more precise cell type annotation.
• Successfully reconstructed GRNs, modeled transcriptional responses to perturbations, and enhanced drug response prediction.

Conclusions:

• RegFormer provides a biologically grounded and scalable framework for single-cell representation learning.
• The model enables deeper mechanistic insights into gene regulation and cellular state transitions.
• It represents a significant advancement in computational approaches for single-cell genomics.