在语境学习的玻璃模型
Yuhao Li1, Ruoran Bai1, Haiping Huang1,2
1Sun Yat-sen University, PMI Lab, School of Physics, Guangzhou 510275, People's Republic of China.
Physical review. E
|August 19, 2025
概括
大型语言模型 (LLM) 展示了无需再培训的上下文学习. 一个新的旋转玻璃模型通过将LLM预训与物理原理和任务多样性联系起来来解释这种新兴的能力.
科学领域:
- 人工智能的人工智能
- 统计物理 统计物理
- 机器学习 机器学习
背景情况:
- 大型语言模型 (LLM) 展示了非凡的上下文学习 (ICL) 能力,在没有明确的再培训的情况下,从提示中预测结果.
- 基于ICL的基本机制和物理解释仍然在很大程度上无法解释,这给该领域带来了重大挑战.
研究的目的:
- 提供一个机械的解释在转换器上下文学习.
- 连接上下文学习现象与统计物理学的既定原则.
主要方法:
- 分析了一种简化的变压器模型,对线性注意力进行了分析.
- 这个模型被映射到一个旋转玻璃模型中,具有实值的旋转.
- 旋转玻璃模型的参数 (合和场) 用于解释预训练期间的数据乱和重量相互作用.
主要成果:
- 旋转玻璃模型阐明了重量参数在预训练期间如何相互作用,解释了没有进一步训练的预测.
- 增加任务多样性被证明可以推动单一实例学习场景中的上下文学习的出现.
- 博尔茨曼分布的趋同到重量参数的唯一解决方案与任务多样性和ICL有关.
结论:
- 拟议的旋转玻璃模型为解释LLM行为提供了一个可处理的理论框架.
- 这项研究提供了关于预训练和任务多样性如何实现上下文学习的见解.
- 这项工作通过基于物理的镜头,为理解大型语言模型令人费解的特性开辟了新的途径.
相关概念视频
Observational Learning
311
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
311
Purposive Learning
206
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
206
Associative Learning
572
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
572
Cognitive Learning
517
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
517
Spin–Spin Coupling Constant: Overview
1.0K
In bromoethane, the three methyl protons are coupled to the two methylene protons that are three bonds away. In accordance with the n+1 rule, the signal from the methyl protons is split into three peaks with 1:2:1 relative intensities. The methylene protons appear as a quartet, with the relative intensities of 1:3:3:1.
Qualitatively, any spin plus-half nucleus polarizes the spins of its electrons to the minus-half state. Consequently, the paired electron in the hydrogen–carbon bond must...
Qualitatively, any spin plus-half nucleus polarizes the spins of its electrons to the minus-half state. Consequently, the paired electron in the hydrogen–carbon bond must...
1.0K
Spin–Spin Coupling: Two-Bond Coupling (Geminal Coupling)
1.1K
Two NMR-active nuclei bonded to a central atom can be involved in geminal or two-bond coupling. Geminal coupling is commonly seen between diastereotopic protons in chiral molecules and unsymmetrical alkenes, among others.
The central atom need not be NMR-active because its electrons are affected by the electron polarization of the spin-active atoms. However, spin information is transmitted less effectively than in one-bond coupling, and 2J values are usually weaker than 1J values. The energy of...
The central atom need not be NMR-active because its electrons are affected by the electron polarization of the spin-active atoms. However, spin information is transmitted less effectively than in one-bond coupling, and 2J values are usually weaker than 1J values. The energy of...
1.1K


