Genomic language models with k-mer tokenization strategies for plant genome annotation and regulatory element...

Recent advances in genomic language models have improved the accuracy of in silico analyses, yet many rely on resource-intensive architectures. In this study, we focus on the impact of k-mer tokenization strategies-specifically varying window sizes (three to eight) and overlap schemes-on the performance of transformer-based genomic language models. Through extensive evaluation across multiple plant genomic tasks, including splice site and alternative polyadenylation site prediction, we show that thoughtful design of the k-mer tokenizer plays a critical role in model performance, often outweighing model scale. In particular, overlap-based tokenization generally enhances performance by preserving local sequence context, while certain non-overlap configurations achieve competitive accuracy with improved computational efficiency in some tasks. Despite using a smaller model, our approach performs on par with the state-of-the-art AgroNT model in many cases. These results emphasize that k-mer tokenization, not merely model size, is a key determinant of success in genomic sequence modeling. Our findings provide practical guidance for designing efficient genomic language models tailored to plant biology.