Tokenization efficiency of current foundational large language models for the Ukrainian language

Area of Science:

• Natural Language Processing
• Computational Linguistics

Background:

• Large language models (LLMs) are increasingly used in multilingual settings.
• Tokenization efficiency is a major challenge for low-resource languages, impacting speed, cost, and performance.
• Ukrainian language models face specific hurdles due to underrepresentation in existing tokenizer vocabularies.

Purpose of the Study:

• To compare the performance of various tokenizers for pretrained LLMs specifically for the Ukrainian language.
• To measure tokenization fertility for state-of-the-art models across general and domain-specific Ukrainian text.
• To investigate the efficacy of a transliteration approach for enhancing Ukrainian tokenization efficiency without data loss.

Main Methods:

• Comparative analysis of multiple tokenizers applied to Ukrainian text.
• Tokenization fertility measurements for current SOTA LLMs.
• Experimental evaluation of a transliteration strategy for improved tokenization.

Main Results:

• Identified significant inefficiencies in current LLM tokenizers for the Ukrainian language.
• Quantified tokenization fertility across different models and Ukrainian language domains.
• Demonstrated potential improvements in tokenization efficiency using a transliteration method.

Conclusions:

• Existing LLM tokenizers present disadvantages for Ukrainian language modeling.
• The study highlights challenges in computational cost and processing speed for low-resource languages like Ukrainian.
• Transliteration offers a viable strategy to mitigate tokenization inefficiencies and improve LLM accessibility for Ukrainian.