在大型语言模型中,下一个令牌预测的定律
1University of Rochester, Rochester, New York 14642, USA.
Physical review. E
|October 21, 2025
概括
我们发现了一个普遍的定律,它决定了大型语言模型 (LLM) 如何学习代码嵌入. 每个层均能提高预测的准确性,为LLM开发和解释提供了洞察力.
科学领域:
- 人工智能的人工智能
- 自然语言处理自然语言处理.
- 机器学习 机器学习
背景情况:
- 大型语言模型 (LLM) 很强大,但往往充当黑子.
- 了解LLM中的内部数据处理对于可靠的预测至关重要.
研究的目的:
- 引入一个量化法,解释在LLMs中嵌入学习的上下文化代币.
- 为预测准确性提供层级贡献的洞察力.
主要方法:
- 在预训练的LLMs中分析中间层表示.
- 嵌入学习用于下一个令牌预测的定量建模.
主要成果:
- 确定了一个精确的定量法规,管理嵌入式学习.
- 每一个层,从下到上,都对预测准确性作出了同等的贡献.
- 这种现象在不同的LLM架构和数据集中是一致的.
结论:
- 发现的法律为LLM内部工作提供了一个普遍的视角.
- 调查结果为LLM开发,扩展,预培训和解释提供了可操作的见解.
相关概念视频
01:03Prediction Intervals
3.3K
The interval estimate of any variable is known as the prediction interval. It helps decide if a point estimate is dependable.
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
3.3K
02:07Improving Translational Accuracy
3.5K
3.5K
02:07Improving Translational Accuracy
14.1K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
14.1K
01:22Language Development
841
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
841
01:26Per-Unit Sequence Models
428
An ideal Y-Y transformer, grounded through neutral impedances, displays per-unit sequence networks akin to those of a single-phase ideal transformer when subjected to balanced positive- or negative-sequence currents. These currents do not produce neutral currents, and their associated voltage drops.
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
428
01:27Language and Cognition
704
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
704