An overview of self-supervised deep learning applications to molecular data

The emergence of high-throughput sequencing technologies has generated unprecedented amounts of molecular data, posing significant challenges for analysis and interpretation. In the past 15 years, deep learning (DL) has revolutionized many data analysis fields. However, a key limitation of most DL approaches is their reliance on massive amounts of data that often require labeling. Self-supervised learning (SSL) uses large-scale unlabeled data to learn meaningful representations for specific tasks, allowing training of the models without labels. While SSL has primarily been applied to fields like natural language processing and medical image analysis, SSL can also be applied to molecular data to learn meaningful representations of molecular sequences that can be used for downstream tasks. Despite the growing interest in SSL applications using molecular data over the recent years, no comprehensive review has been published on this topic, so far, making it timely to address. This paper aims to provide researchers in DL and bioinformatics with a clear view of SSL omics applications to foster future work in the domain. This review examines the principles of SSL, such as foundation models, and it discusses the application of SSL to various omics data types, summarizes information from 17 studies, and categorizes applications by data type, detailing common tasks, model architectures, and repositories. Key applications such as DNABERT and Nucleotide Transformer are highlighted, demonstrating the contributions of SSL in understanding gene regulation. Future directions for SSL in omics are outlined, emphasizing the potential for integrating multi-omics data and developing more sophisticated pretext tasks.