测试自注意的旋转浴视图:对GPT-2变压器的哈密尔顿分析
Satadeep Bhattacharjee1, Seung-Cheol Lee2
1Indo-Korea Science and Technology Center (IKST), Bangalore, India.
Physical review. E
|February 20, 2026
概括
这项研究验证了基于物理的旋转系统模型,用于大语言模型的注意力. 经验证据表明,这个模型准确地预测了代币主导地位,提供了新的AI解释性见解.
科学领域:
- 人工智能的人工智能
- 凝聚物质物理学 凝聚物质物理学
- 计算语言学 计算语言学
背景情况:
- 大型语言模型 (LLM) 表现出复杂的行为,如重复和偏见.
- 现有的解释性方法往往缺乏第一原则的基础.
- 一个新的基于物理的框架模型将LLM注意力作为一个旋转系统.
研究的目的:
- 为了经验验证LLM注意力机制的旋转浴类比.
- 从基于物理的模型中推导出用于令牌选择的预测标准.
- 为了研究旋转系统动态和LLM输出之间的因果关系.
主要方法:
- 从GPT-2模型中提取了查询键重量矩阵.
- 对于每个注意力头的有效哈密尔顿推导.
- 开发了分析阶段边界和逻辑差距标准.
- 对事实回忆提示和有针对性的消去进行了系统的评估.
主要成果:
- 在理论逻辑差距和经验令牌排名之间发现了一个强大的负相关性 (r≈-0.70).
- 抑制预测的主导注意力头因果转移了输出概率.
- 旋转浴的类比得到了强有力的经验证据的支持,在一个生产级的LLM.
结论:
- 旋转浴类比为LLM注意力提供了一个强有力的,基于物理的解释.
- 衍生出的标准为令牌选择提供了预测能力.
- 这项工作将凝聚物质物理学和人工智能联系起来,为新的生成模型铺平了道路.
相关概念视频
Gyroscope
A gyroscope is defined as a spinning disk in which the axis of rotation is free to assume any orientation. When spinning, the orientation of the spin axis is unaffected by the orientation of the body that encloses it. The body or vehicle enclosing the gyroscope can be moved from place to place, while the orientation of the spin axis remains the same. This makes gyroscopes very useful in navigation, especially where magnetic compasses cannot be used, such as in crewed and crewless spacecraft,...
Spin–Spin Coupling Constant: Overview
In bromoethane, the three methyl protons are coupled to the two methylene protons that are three bonds away. In accordance with the n+1 rule, the signal from the methyl protons is split into three peaks with 1:2:1 relative intensities. The methylene protons appear as a quartet, with the relative intensities of 1:3:3:1.
Qualitatively, any spin plus-half nucleus polarizes the spins of its electrons to the minus-half state. Consequently, the paired electron in the hydrogen–carbon bond must have a...
Qualitatively, any spin plus-half nucleus polarizes the spins of its electrons to the minus-half state. Consequently, the paired electron in the hydrogen–carbon bond must have a...
Spin–Spin Coupling: Two-Bond Coupling (Geminal Coupling)
Two NMR-active nuclei bonded to a central atom can be involved in geminal or two-bond coupling. Geminal coupling is commonly seen between diastereotopic protons in chiral molecules and unsymmetrical alkenes, among others.
The central atom need not be NMR-active because its electrons are affected by the electron polarization of the spin-active atoms. However, spin information is transmitted less effectively than in one-bond coupling, and 2J values are usually weaker than 1J values. The energy of...
The central atom need not be NMR-active because its electrons are affected by the electron polarization of the spin-active atoms. However, spin information is transmitted less effectively than in one-bond coupling, and 2J values are usually weaker than 1J values. The energy of...
Energy Losses in Transformers
In an ideal transformer, it is assumed that there are no energy losses, and, hence, all the power at the primary winding is transferred to the secondary winding. However, in reality, the transformers always have some energy losses, and, hence, the output power obtained at the secondary winding is less than the input power at the primary winding due to energy losses.
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the copper windings...
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the copper windings...
Three-Winding Transformers
Three identical single-phase transformers can be configured to form a three-phase transformer connection, which involves high-voltage and low-voltage windings. The high-voltage windings are denoted by capital letters A-B-C, while the low-voltage windings are labeled with lowercase letters a-b-c, representing their respective phases. This notation helps distinguish between the high and low voltage sides of the transformer.
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
Multimachine Stability
Multimachine stability analysis is crucial for understanding the dynamics and stability of power systems with multiple synchronous machines. The objective is to solve the swing equations for a network of M machines connected to an N-bus power system.
In analyzing the system, the nodal equations represent the relationship between bus voltages, machine voltages, and machine currents. The nodal equation is given by:
In analyzing the system, the nodal equations represent the relationship between bus voltages, machine voltages, and machine currents. The nodal equation is given by:


