简单的层次语言中的缩放规律和表示学习:变压器与卷积架构对比
Francesco Cagnetta1, Alessandro Favero2, Antonio Sclocchi3
1Scuola Internazionale Superiore di Studi Avanzati, (SISSA), Via Bonomea 265, 34136 Trieste, Italy.
Physical review. E
|January 21, 2026
概括
神经语言模型通过下一个令牌预测来学习语言结构. 卷积网络的性能优于变压器,因为它与数据结构的架构对齐,澄清了神经缩放规律.
科学领域:
- 计算语言学计算语言学
- 机器学习理论机器学习理论
- 深度学习架构是一种深度学习架构.
背景情况:
- 神经语言模型 (NLM) 在下一个令牌预测任务上受训.
- 了解NLM如何获得语言结构是一个关键的研究问题.
- 之前的工作建立了一个基于数据相关性的表示学习理论.
研究的目的:
- 在合成数据上训练的NLM来导出理论缩放规律.
- 调查模型架构对学习语言结构的影响.
- 扩展现有的表示学习理论,以解释架构差异.
主要方法:
- 开发了神经网络性能的理论缩放规律.
- 使用来自随机等级模型 (RHM) 的合成数据集.
- 扩展了表示学习理论,以纳入架构变异.
主要成果:
- 卷积网络显示出比变压器模型更快的性能扩展.
- 确定了神经缩放规律中的架构偏差.
- 关于网络性能的经验验证的预测.
结论:
- 模型架构显著影响NLM学习语言结构的方式.
- 网络架构与数据的统计属性之间的对齐至关重要.
- 结果澄清了在NLM中架构,数据和表示学习之间的相互作用.
相关概念视频
Convolution Properties II
582
The important convolution properties include width, area, differentiation, and integration properties.
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
582
Second Law of Thermodynamics
67.9K
The Second Law of Thermodynamics states that entropy, or the amount of disorder in a system, increases each time energy is transferred or transformed. Each energy transfer results in a certain amount of energy that is lost—usually in the form of heat—that increases the disorder of the surroundings. This can also be demonstrated in a classic food web. Herbivores harvest chemical energy from plants and release heat and carbon dioxide into the environment. Carnivores harvest the...
67.9K
First Law of Thermodynamics
80.3K
The First Law of Thermodynamics states that energy cannot be created or destroyed, only transformed. This can be demonstrated within a classic food web where light energy from the sun is harnessed as radiant energy by plants, converted into chemical energy, and stored as complex carbohydrates. The vegetation is then consumed by animals and during the digestion process, the sugars release energy as heat. The sugars also produce chemical energy that either gets used up doing work, stored in...
80.3K
First Law of Thermodynamics
40.4K
Energy Conservation
40.4K
Language
892
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
892
Gas Laws: Boyle's, Gay-Lussac, Charles', Avogadro's, and Ideal Gas Law
76.6K
Through experiments, scientists established the mathematical relationships between pairs of variables, such as pressure and temperature, pressure and volume, volume and temperature, and volume and moles, that hold for an ideal gas.
76.6K


