Large-scale foundation model on single-cell transcriptomics

Area of Science:

• Computational biology
• Genomics
• Bioinformatics

Background:

• Foundation models have revolutionized NLP and related fields.
• Developing similar models for single-cell transcriptomics presents significant challenges.
• Understanding cellular 'languages' is crucial for biomedical advancements.

Purpose of the Study:

• To develop a large-scale foundation model, scFoundation (xTrimoscFoundationα), for single-cell transcriptomic data analysis.
• To leverage a transformer-like architecture and novel pretraining tasks for capturing gene interdependencies.
• To establish a versatile tool for diverse single-cell omics applications.

Main Methods:

• Developed scFoundation, a 100-million-parameter model trained on over 50 million human single-cell transcriptomic profiles.
• Utilized an asymmetric transformer-like architecture designed for complex gene context relations.
• Employed specific pretraining tasks tailored for single-cell data characteristics.

Main Results:

• scFoundation demonstrated state-of-the-art performance across multiple single-cell analysis tasks.
• Achieved high accuracy in gene expression enhancement and cell type annotation.
• Showcased effectiveness in predicting tissue and single-cell drug responses and perturbations.

Conclusions:

• scFoundation serves as a powerful foundation model for single-cell transcriptomics.
• The model's architecture and pretraining enable robust analysis of gene expression data.
• It offers significant potential to accelerate biomedical research and drug discovery.