The Impact of Tokenizer Selection in Genomic Language Models

Area of Science:

• Computational biology
• Genomic data analysis
• Artificial intelligence in genomics

Background:

• Genomic language models (GLMs) are emerging tools for genetic sequence analysis.
• GLMs use various tokenization methods, including character, k-mer, and byte-pair encoding (BPE).
• Genomic sequences present unique challenges compared to natural language, impacting tokenization strategies.

Purpose of the Study:

• To investigate the impact of different tokenization methods on GLM performance.
• To evaluate tokenization strategies across diverse genomic classification tasks.
• To compare character tokenization with sub-word tokenization (BPE) in the Mamba model.

Main Methods:

• Evaluated downstream performance of GLMs on 44 classification fine-tuning tasks.
• Conducted a direct comparison between byte-pair encoding and character tokenization within the Mamba state-space model.
• Assessed model performance based on task-specific metrics.

Main Results:

• Character tokenization outperformed sub-word methods on tasks requiring nucleotide-level resolution, such as splice site prediction and promoter detection.
• Byte-pair encoding demonstrated superior performance in SARS-CoV-2 variant classification.
• Limited statistically significant differences were observed between tokenization methods for most other downstream tasks.

Conclusions:

• The choice of tokenization significantly impacts GLM performance, particularly for nucleotide-level genomic tasks.
• Character tokenization is a robust choice for tasks demanding high resolution.
• Further research is needed to optimize tokenization for diverse genomic applications and model architectures.