Semi-supervised learning with pseudo-labeling compares favorably with large language models for regulatory sequence...

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Deep learning offers biological insights for non-coding single nucleotide polymorphisms (SNPs) from genome-wide association studies (GWAS).
• Supervised learning methods are limited by the scarcity of functional data for DNA sequences.
• Mammalian DNA sequence data is abundant but often lacks associated functional information.

Purpose of the Study:

• To develop a novel semi-supervised learning (SSL) approach to overcome limitations of supervised learning in predicting molecular processes from DNA.
• To enhance model pre-training by effectively utilizing large amounts of unlabeled DNA sequences.
• To improve predictive performance, especially in scenarios with limited functional data, such as predicting transcription factor binding.

Main Methods:

• Proposed a novel semi-supervised learning (SSL) method based on pseudo-labeling for pre-training models on unlabeled DNA sequences.
• Incorporated principles from the Noisy Student algorithm to refine pseudo-labeled data confidence during pre-training.
• The flexible approach allows training of various neural network architectures, including advanced models.

Main Results:

• The SSL approach demonstrated significant improvements in predictive performance compared to standard supervised learning across various tasks.
• The method showed particular effectiveness for predicting transcription factor binding sites, even with very limited training data.
• Small models trained using SSL achieved performance comparable to or exceeding that of large language models like DNABERT2.

Conclusions:

• Semi-supervised learning, particularly with pseudo-labeling and Noisy Student principles, is a powerful strategy for deep learning in genomics.
• This approach effectively leverages abundant unlabeled DNA sequences to enhance predictive modeling for molecular processes.
• SSL provides a flexible and high-performing alternative to supervised learning, especially when functional genomic data is scarce.