Self-distillation improves self-supervised learning for DNA sequence inference

Area of Science:

• Genomics
• Bioinformatics
• Machine Learning

Background:

• Self-supervised learning (SSL) enhances prediction accuracy in many fields.
• Current SSL methods for DNA sequences often neglect multi-sequence statistical information.
• Existing approaches primarily focus on masked language modeling of individual sequences.

Purpose of the Study:

• To develop an advanced SSL model for DNA sequences that captures both individual sequence context and population-level distributional data.
• To overcome the limitations of existing SSL methods in genomics by incorporating multi-sequence statistics.

Main Methods:

• Developed a deep neural network with collaborative 'student' and 'teacher' subnetworks.
• Employed masked learning on nucleotides within the student subnetwork.
• Utilized exponential moving average for parameter adaptation between subnetworks.
• Incorporated contrastive learning on augmented sequence representations for both subnetworks.
• Implemented a self-distillation process to integrate contextual and distributional information.

Main Results:

• Pretrained the model using the human reference genome.
• Applied the model to 20 downstream inference tasks.
• Demonstrated significant improvements in inference performance across most tasks.
• The novel approach effectively assimilated contextual and distributional genomic data.

Conclusions:

• The proposed SSL method significantly enhances prediction accuracy for DNA sequences.
• The model's ability to learn from both individual sequences and the overall population is key to its success.
• This approach offers a promising direction for advancing computational genomics and bioinformatics.