Understanding the Natural Language of DNA using Encoder-Decoder Foundation Models with Byte-level Precision.

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Genomic sequence analysis traditionally relies on tokenization methods.
• Existing foundation models often use encoder-only or decoder-only architectures.
• Byte-level precision is crucial for accurate analysis of DNA sequences.

Purpose of the Study:

• Introduce the Ensemble Nucleotide Byte-level Encoder-Decoder (ENBED) foundation model.
• Develop an efficient, generalizable sequence-to-sequence model for genomic tasks.
• Demonstrate ENBED's superiority over existing state-of-the-art methods.

Main Methods:

• Utilized an encoder-decoder Transformer architecture with sub-quadratic attention.
• Pre-trained the foundation model using Masked Language Modeling on reference genome sequences.
• Applied the model to downstream tasks including sequence annotation and mutation generation.

Main Results:

• Achieved significant improvements in identifying enhancers, promoters, and splice sites.
• Successfully recognized sequences with base call errors and indels at byte-level precision.
• Demonstrated state-of-the-art performance in biological function annotation and viral mutation generation.

Conclusions:

• ENBED offers a powerful and efficient approach for byte-level genomic sequence analysis.
• The encoder-decoder architecture generalizes previous genomic models.
• ENBED significantly advances the state-of-the-art in multiple critical bioinformatics tasks.