Transfer learning enables predictions in network biology.

Area of Science:

• Network biology
• Genomics
• Computational biology

Background:

• Mapping gene networks typically requires extensive transcriptomic data, limiting discoveries in rare diseases or hard-to-access tissues.
• Transfer learning, using models pre-trained on large datasets, has shown success in other fields by enabling fine-tuning for specific tasks with less data.

Purpose of the Study:

• To develop a deep learning model, Geneformer, for context-specific predictions in network biology, especially when transcriptomic data is limited.
• To leverage transfer learning by pretraining Geneformer on a large corpus of single-cell transcriptomes.

Main Methods:

• Developed Geneformer, a context-aware, attention-based deep learning model.
• Pretrained Geneformer on approximately 30 million single-cell transcriptomes in a self-supervised manner.
• Fine-tuned Geneformer on various downstream tasks related to chromatin and network dynamics.

Main Results:

• Geneformer learned fundamental network dynamics and hierarchy during pretraining, encoded in its attention weights.
• Fine-tuning Geneformer on limited data consistently improved predictive accuracy across diverse tasks.
• Applied to disease modeling, Geneformer identified potential therapeutic targets for cardiomyopathy using limited patient data.

Conclusions:

• Geneformer is a powerful pretrained deep learning model for network biology applications with limited data.
• Fine-tuning Geneformer can accelerate the discovery of key gene network regulators.
• The model shows promise for identifying candidate therapeutic targets in disease modeling.