seqLens: Optimizing Language Models for Genomic Predictions

Area of Science:

• Computational Biology
• Genomics
• Machine Learning
• Evolutionary Biology

Background:

• Language modeling offers a novel approach to understanding genomic sequence variation and evolutionary patterns.
• Existing language models face computational challenges in tokenization and architecture for diverse genomic data across evolutionary scales.

Purpose of the Study:

• To investigate key elements of genomic language modeling (gLM), including tokenization, pretraining datasets, and model architecture.
• To apply and evaluate gLMs for identifying evolutionary genomic features and improving genome annotations.

Main Methods:

• Gathered two distinct pretraining datasets comprising prokaryotic and eukaryotic reference genomes.
• Trained five byte-pair encoding tokenizers and pretrained 52 gLMs, comparing various architectures and hyperparameters.
• Introduced seqLens, a novel model architecture based on disentangled attention with relative positional encoding.

Main Results:

• The seqLens model family demonstrated superior performance in 13 of 19 benchmarking phenotypic predictions compared to similar-sized models.
• Relevant pretraining data significantly enhanced gLM performance, while larger tokenizer vocabularies negatively impacted generalization.
• Alternative pooling techniques improved classification accuracy, and gLMs showed capability in discerning evolutionary relationships.

Conclusions:

• Optimizing tokenization strategies and pretraining datasets is crucial for effective genomic language modeling.
• The developed gLMs can successfully identify diverse evolutionary genomic features and aid in genome annotation.
• Findings provide a foundation for advancing language models in evolutionary genomics research.