Continually adapting pre-trained language model to universal annotation of single-cell RNA-seq data

Area of Science:

• Computational biology
• Genomics
• Bioinformatics

Background:

• Cell-type annotation of single-cell RNA sequencing (scRNA-seq) data is crucial for biomedical research.
• Existing tools struggle with continuously emerging scRNA-seq data, necessitating adaptive models.
• Transformer-based pre-trained language models offer powerful data integration and interpretability for single-cell biology.

Purpose of the Study:

• To develop a universal cell-type annotation tool that continuously learns from new scRNA-seq data.
• To address the challenge of catastrophic forgetting in continual learning models for scRNA-seq annotation.
• To enable the expansion of cell-type libraries and identification of novel cell types.

Main Methods:

• Proposed CANAL, a tool that fine-tunes pre-trained language models on emerging labeled scRNA-seq data.
• Implemented an experience replay schema with a dynamic, class-balanced example bank to retain input knowledge.
• Utilized representation knowledge distillation to preserve output knowledge from previous model states.
• Designed a framework to incorporate new cell types during fine-tuning and testing.

Main Results:

• CANAL effectively alleviates catastrophic forgetting in both model inputs and outputs.
• The dynamic example bank aids in consolidating patterns, especially for rare cell types.
• Representation knowledge distillation ensures knowledge preservation across training stages.
• CANAL demonstrated versatility and high interpretability in experiments with diverse data streams.

Conclusions:

• CANAL provides a robust solution for continuous cell-type annotation of scRNA-seq data.
• The tool can dynamically expand its cell-type annotation library and identify novel cells.
• CANAL represents a significant advancement in adapting machine learning models to the evolving landscape of single-cell genomics.