自己注意のスピンバスの見方をテストする:GPT-2変圧器のハミルトン解析
Satadeep Bhattacharjee1, Seung-Cheol Lee2
1Indo-Korea Science and Technology Center (IKST), Bangalore, India.
Physical review. E
|February 20, 2026
まとめ
この研究は,物理に基づいたスピンシステムモデルを,大きな言語モデルの注意のために検証しています. 経験的証拠は,このモデルがトークンの優位性を正確に予測し,新しいAI解釈可能性の洞察を提供することを示しています.
科学分野:
- 人工知能 (AI) とは,人工知能 (AI) のことです.
- 凝縮された物質の物理学
- コンピュータ言語学 コンピュータ言語学
背景:
- 大型言語モデル (LLM) は,繰り返しやバイアスなどの複雑な行動を示す.
- 既存の解釈方法には,しばしば第一原理の根拠が欠けている.
- 新しい物理ベースのフレームワークが,スピンシステムとしてLLMの注意をモデル化しています.
研究 の 目的:
- LLMの注意力メカニズムについて,スピン・バス・アナロギーを実証的に検証する.
- 物理ベースのモデルからトークン選択の予測基準を導き出す.
- スピンシステムのダイナミクスとLLMの出力との因果関係を調査する.
主な方法:
- GPT-2モデルからクエリキー重量行列を抽出した.
- 各注意点の有効なハミルトン値を導き出した.
- 分析段階の境界とロギットギャップの基準を開発した.
- 事実を想起させるプロンプトと標的の消去に関する体系的な評価を実施した.
主要な成果:
- 理論的なロギットギャップと経験的トークンランキングの間で強い負の相関 (r≈-0.70) が発見されました.
- 予測された支配的注意を抑制する頭は因果的に出力確率をシフトした.
- スピン・バス・アナロギーは,プロダクション・グレードのLLMにおける強力な経験的証拠によって支持されました.
結論:
- スピン・バス・アナロジーは,LLMの注意を物理的に根拠づけられた強力な説明を提供します.
- 派生した基準は,トークンの選択のための予測力を提供します.
- この研究は,凝縮物質物理学とAIの架け橋となり,新しい生成モデルへの道を開く.
関連する概念動画
01:02Gyroscope
A gyroscope is defined as a spinning disk in which the axis of rotation is free to assume any orientation. When spinning, the orientation of the spin axis is unaffected by the orientation of the body that encloses it. The body or vehicle enclosing the gyroscope can be moved from place to place, while the orientation of the spin axis remains the same. This makes gyroscopes very useful in navigation, especially where magnetic compasses cannot be used, such as in crewed and crewless spacecraft,...
01:08Spin–Spin Coupling Constant: Overview
In bromoethane, the three methyl protons are coupled to the two methylene protons that are three bonds away. In accordance with the n+1 rule, the signal from the methyl protons is split into three peaks with 1:2:1 relative intensities. The methylene protons appear as a quartet, with the relative intensities of 1:3:3:1.
Qualitatively, any spin plus-half nucleus polarizes the spins of its electrons to the minus-half state. Consequently, the paired electron in the hydrogen–carbon bond must have a...
Qualitatively, any spin plus-half nucleus polarizes the spins of its electrons to the minus-half state. Consequently, the paired electron in the hydrogen–carbon bond must have a...
01:20Spin–Spin Coupling: Two-Bond Coupling (Geminal Coupling)
Two NMR-active nuclei bonded to a central atom can be involved in geminal or two-bond coupling. Geminal coupling is commonly seen between diastereotopic protons in chiral molecules and unsymmetrical alkenes, among others.
The central atom need not be NMR-active because its electrons are affected by the electron polarization of the spin-active atoms. However, spin information is transmitted less effectively than in one-bond coupling, and 2J values are usually weaker than 1J values. The energy of...
The central atom need not be NMR-active because its electrons are affected by the electron polarization of the spin-active atoms. However, spin information is transmitted less effectively than in one-bond coupling, and 2J values are usually weaker than 1J values. The energy of...
01:21Energy Losses in Transformers
In an ideal transformer, it is assumed that there are no energy losses, and, hence, all the power at the primary winding is transferred to the secondary winding. However, in reality, the transformers always have some energy losses, and, hence, the output power obtained at the secondary winding is less than the input power at the primary winding due to energy losses.
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the copper windings...
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the copper windings...
01:19Three-Winding Transformers
Three identical single-phase transformers can be configured to form a three-phase transformer connection, which involves high-voltage and low-voltage windings. The high-voltage windings are denoted by capital letters A-B-C, while the low-voltage windings are labeled with lowercase letters a-b-c, representing their respective phases. This notation helps distinguish between the high and low voltage sides of the transformer.
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
01:25Multimachine Stability
Multimachine stability analysis is crucial for understanding the dynamics and stability of power systems with multiple synchronous machines. The objective is to solve the swing equations for a network of M machines connected to an N-bus power system.
In analyzing the system, the nodal equations represent the relationship between bus voltages, machine voltages, and machine currents. The nodal equation is given by:
In analyzing the system, the nodal equations represent the relationship between bus voltages, machine voltages, and machine currents. The nodal equation is given by: