Transformers for single-cell RNA sequencing: a survey

Transformers have demonstrated remarkable success in the field of deep learning, attracting significant attention from researchers and driving investigations into their applications in biomedical data analysis. Single-cell RNA sequencing (scRNA-seq) is an emerging resource important for research on disease progression and tumor microenvironments. However, scRNA-seq datasets are characterized by challenges including sparseness, high-dimensionality, large-scale, and sensitivity to batch effects. These features often necessitate substantial computational resources and can drag the performance of conventional neural networks, which frequently fail to deliver satisfactory results. The application of Transformers to single-cell sequencing is promising to solve these problems, and due to its self-attention mechanism and transfer learning paradigm, Transformers have significantly improved its model performance. This survey aims to provide a comprehensive overview of Transformers applied to scRNA-seq. It systematically analyzes the construction and capacity of Transformers in two fields: (i) Transformers designed for specific tasks, and (ii) Transformers developed to address multiple downstream tasks, often referred to as foundation models. In addition, this work examines Transformer models from the perspectives of performance, computational efficiency, interpretability, and scalability, and outlines potential avenues for future research, including emerging efforts on other omics layers beyond scRNA-seq. By offering a detailed analysis, we aim to provide a practical resource for both newcomers and experienced researchers, push forward the development of Transformer-based models for single-cell sequencing, address current challenges, and inspire further progress in the field.