目前乌克兰语言基础大型语言模型的代币化效率
Daniil Maksymenko1, Oleksii Turuta2
1Department of Artificial Intelligence, Kharkiv National University of Radio Electronics, Kharkiv, Ukraine.
Frontiers in artificial intelligence
|August 29, 2025
概括
这项研究评估了乌克兰语的大型语言模型 (LLM) 的代币化工具,发现当前的模型对低资源语言无效. 一种转写方法有望提高乌克兰语模拟效率.
科学领域:
- 自然语言处理
- 计算语言学
背景情况:
- 大型语言模型 (LLM) 在多语言环境中越来越多地使用.
- 代币化效率是低资源语言的主要挑战,影响速度,成本和性能.
- 由于在现有的代币化词汇中代表性不足,乌克兰语言模型面临着特殊的障碍.
研究的目的:
- 针对乌克兰语的预训练LLM进行各种代币化器的性能比较.
- 测量通用和特定领域的乌克兰文本中最先进的模型的代币化生长率.
- 调查转写方法的有效性,以提高乌克兰的代币化效率而不会丢失数据.
主要方法:
- 对乌克兰文本应用的多个tokenizers的比较分析.
- 目前SOTALLM的代币化生育率测量
- 对改进代币化的转写策略进行实验性评估.
主要成果:
- 确定了乌克兰语言的现有LLM代币化器的显著低效率.
- 在不同模型和乌克兰语言领域量化代币化生长率.
- 使用转写方法证明了代币化效率的潜在改进.
结论:
- 现有的LLM代币对乌克兰语言建模具有缺点.
- 这项研究强调了乌克兰语等资源较低语言的计算成本和处理速度的挑战.
- 转字提供了一种可行的策略来缓解代币化效率低下,并提高乌克兰的LLM可访问性.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
570
09:09Foreign Accent and Forensic Speaker Identification in Voice Lineups: The Influence of Acoustic Features Based on Prosody
Published on: September 27, 2024
522
相关概念视频
02:07Improving Translational Accuracy
11.8K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
11.8K
01:22Language Development
444
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
444
01:27Language and Cognition
438
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
438
01:24Components of Language
388
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
388
01:16Language
422
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
422
01:26Per-Unit Sequence Models
116
An ideal Y-Y transformer, grounded through neutral impedances, displays per-unit sequence networks akin to those of a single-phase ideal transformer when subjected to balanced positive- or negative-sequence currents. These currents do not produce neutral currents, and their associated voltage drops.
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
116