The impact of tokenizer selection in genomic language models

Area of Science:

• Computational Biology
• Genomics
• Machine Learning

Background:

• Genomic language models (GLMs) offer novel approaches for genetic sequence analysis.
• GLMs employ diverse tokenization strategies, including character, k-mer, and byte-pair encoding (BPE).
• Genomic data presents unique challenges for tokenization compared to natural or protein language due to low variability and complex features.

Purpose of the Study:

• To investigate the impact of different tokenization strategies on GLM performance.
• To compare character tokenization against sub-word methods like BPE within GLMs.
• To evaluate tokenization effects across a wide range of downstream genomic classification tasks.

Main Methods:

• Evaluated downstream performance of GLMs using forty-four classification fine-tuning tasks.
• Conducted a direct comparison of byte-pair encoding and character tokenization in the Mamba state-space model.
• Benchmarked tokenization methods on tasks requiring nucleotide-level resolution and broader genomic classification.

Main Results:

• Character tokenization outperformed sub-word methods on nucleotide-specific tasks, such as splice site prediction and promoter detection.
• Byte-pair encoding demonstrated superior performance for SARS-CoV-2 variant classification.
• Limited statistically significant differences were observed between tokenization methods for most other downstream tasks.

Conclusions:

• Character tokenization is highly effective for genomic tasks demanding nucleotide-level precision.
• The choice of tokenization method in GLMs should be guided by the specific requirements of the downstream application.
• Further research into optimal tokenization for diverse genomic data types is warranted.