Pretraining model for biological sequence data.

Area of Science:

• Bioinformatics and Computational Biology
• Genomics and Proteomics
• Machine Learning in Life Sciences

Background:

• High-throughput sequencing generates vast biological sequence data, vital for understanding life and disease.
• The COVID-19 pandemic highlighted the importance of biological sequence data for diagnostics and therapeutics.
• Pretraining models from natural language processing offer efficient feature extraction and improved performance for biological sequence analysis.

Purpose of the Study:

• To provide a comprehensive review of pretraining models for biological sequence data.
• To introduce biological sequences, datasets, and categorize existing pretraining models.
• To discuss applications, novel methods, and future challenges in the field.

Main Methods:

• Systematic review and categorization of pretraining models (CNN, word2vec, LSTM, Transformer).
• Introduction of biological sequences, datasets, and their accessibility.
• Presentation of applications and a novel pretraining scheme for protein sequences.

Main Results:

• Biological sequence data is increasingly accessible, driving advancements in disease detection and drug discovery.
• Pretraining models effectively embed and extract features from large biological sequence corpora.
• Various pretraining models, including CNN, word2vec, LSTM, and Transformer, are summarized and categorized.

Conclusions:

• Pretraining models are essential for advancing biological sequence data analysis and its applications.
• Novel pretraining schemes and benchmarks are emerging for specific biological sequences like proteins.
• Addressing current challenges and exploring future directions will further enhance the utility of pretraining models.