scLong: a billion-parameter foundation model for capturing long-range gene context in single-cell transcriptomics

Area of Science:

• Genomics
• Computational Biology
• Bioinformatics

Background:

• Single-cell RNA sequencing (scRNA-seq) provides high-resolution gene expression data, revealing cellular heterogeneity.
• Existing foundation models for scRNA-seq data often overlook lowly expressed genes and external biological knowledge.
• Analyzing complex gene interactions is crucial for understanding cellular functions and disease mechanisms.

Purpose of the Study:

• To introduce scLong, a large-scale foundation model for scRNA-seq data analysis.
• To enable comprehensive gene expression modeling, including lowly expressed and unexpressed genes.
• To integrate external gene knowledge for enhanced biological context and predictive power.

Main Methods:

• Pretraining a billion-parameter foundation model (scLong) on 48 million cells.
• Implementing self-attention across all 28,000 human genes to capture long-range dependencies.
• Integrating Gene Ontology knowledge using a graph convolutional network.

Main Results:

• scLong demonstrates superior performance compared to state-of-the-art models across various tasks.
• The model effectively captures dependencies involving lowly expressed and unexpressed genes.
• scLong shows strong capabilities in predicting transcriptional responses, cancer drug efficacy, and gene regulatory networks.

Conclusions:

• scLong represents a significant advancement in foundation models for scRNA-seq data analysis.
• The model's ability to process all genes and integrate external knowledge enhances biological insights.
• scLong has broad applications in understanding gene regulation, disease mechanisms, and therapeutic development.