Understanding the natural language of DNA using encoder-decoder foundation models with byte-level precision.

Area of Science:

• Genomics
• Bioinformatics
• Machine Learning

Background:

• Genomic sequence analysis traditionally relies on tokenization methods.
• Existing models often use encoder-only or decoder-only architectures.
• Byte-level precision in DNA sequence analysis is crucial for detailed insights.

Purpose of the Study:

• Introduce the Ensemble Nucleotide Byte-level Encoder-Decoder (ENBED) foundation model.
• Develop an efficient sequence-to-sequence transformation model for genomic data.
• Generalize and improve upon existing genomic foundation models.

Main Methods:

• Utilized an encoder-decoder Transformer architecture with subquadratic attention.
• Pretrained the model using Masked Language Modeling on reference genome sequences.
• Applied the model to downstream tasks including functional annotation and error detection.

Main Results:

• Achieved state-of-the-art performance in identifying enhancers, promotors, and splice sites.
• Successfully recognized sequences with base call mismatches and insertion/deletion errors.
• Demonstrated significant improvements in biological function annotation and viral mutation generation.

Conclusions:

• ENBED offers superior byte-level precision for genomic sequence analysis.
• The foundation model generalizes effectively across diverse downstream tasks.
• ENBED represents a significant advancement in computational genomics.