Contrastive pre-training for sequence based genomics models

Area of Science:

• Genomics
• Deep Learning
• Bioinformatics

Background:

• Deep learning is increasingly used in genomics.
• Complex models require substantial data or strategic initialization for optimal performance.

Purpose of the Study:

• Introduce cGen, a novel unsupervised, model-agnostic contrastive pre-training method for sequence-based models.
• Improve deep learning model performance in genomics, particularly in data-scarce scenarios.

Main Methods:

• cGen uses unsupervised contrastive pre-training to learn intrinsic genome features.
• It initializes model weights, reducing the need for large datasets.
• The method is model-agnostic and makes no assumptions about genomic structure.

Main Results:

• Embeddings from unsupervised cGen are informative for gene expression prediction.
• Learned sequence features enable meaningful clustering.
• cGen enhances performance in chromatin profiling prediction and gene expression tasks.

Conclusions:

• cGen improves deep learning model performance in genomics without architecture modification.
• This method is particularly beneficial for applications with limited data availability.
• Unsupervised pre-training offers a powerful strategy for advancing genomic deep learning.