在具有层规范化的变压器架构中,从自我注意力动态中产生的平均场模型的分析
Martin Burger1,2, Samira Kabri1, Yury Korolev3
1Helmholtz Imaging, Deutsches Elektronen-Synchrotron, Hamburg, Germany.
概括
这项研究数学分析了具有自我注意力和层规范化的变压器架构. 我们开发了一个梯度流框架,以严格研究这些动态,并提供对它们的数学特性的见解.
科学领域:
- 数学 数学 是一个数学.
- 计算机科学 计算机科学
- 数据科学数据科学数据科学
背景情况:
- 在深度学习中广泛使用的变压器架构采用了具有层规范化的自我注意机制.
- 在这些架构中观察到的数据模式,例如集群或均分布,会带来复杂的数学挑战.
- 了解基础数学原理对于推进人工智能和数据科学应用至关重要.
研究的目的:
- 为变压器架构提供严格的数学分析,重点关注自我注意力和层正常化.
- 调查自我注意力动态的数学特性,特别是关于单位球上的概率测量.
- 在这种情况下,开发一个分析梯度流的框架,并探索动态的静止点.
主要方法:
- 在单位球上的概率测量空间中,用专门的度量表达一个梯度流.
- 数学问题的分析类似于聚合方程,适应球体动力学和特定的相互作用能量.
- 在瓦瑟斯坦几何学中通过相互作用能量的透视来研究静止点.
主要成果:
- 在自我注意力机制的背景下,建立了一个严格的框架来研究梯度流.
- 在变压器架构中观察到的模式引起的数学问题提供了部分答案.
- 分析了自我注意力动态的静止点,将它们与能量最小化和最大化器联系起来.
结论:
- 这项研究为变压器架构和自我注意力机制提供了新的数学视角.
- 开发的梯度流框架为理解这些复杂的动态提供了严格的方法.
- 这些发现有助于数据科学和部分微分方程的数学基础.
更多相关视频
08:45Mapping Cortical Dynamics Using Simultaneous MEG/EEG and Anatomically-constrained Minimum-norm Estimates: an Auditory Attention Example
Published on: October 24, 2012
14.6K
08:51Statistical Modelling of Cortical Connectivity Using Non-invasive Electroencephalograms
Published on: November 1, 2019
5.6K
相关概念视频
01:25Transformers with Off-Nominal Turns Ratios
142
In scenarios involving parallel transformers with disparate ratings, developing per-unit models requires accommodating off-nominal turns ratios. This situation arises when the selected base voltages are not proportional to the transformer’s voltage ratings. Consider a transformer where the rated voltages are related by the term a. If the chosen voltage bases satisfy a relationship involving term b, term c is defined as the ratio of these bases. This ratio is then substituted into the...
142
01:28Equivalent Circuits for Practical Transformers
401
The practical equivalent circuits of single-phase two-winding transformers exhibit significant deviations from their idealized versions due to the inherent properties of winding resistance and finite core permeability. These properties result in real and reactive power losses, affecting the transformer's performance. Understanding these deviations is crucial for designing more efficient transformers.
In a practical transformer, each winding exhibits resistance and leakage reactance. The...
In a practical transformer, each winding exhibits resistance and leakage reactance. The...
401
01:19Three-Winding Transformers
216
Three identical single-phase transformers can be configured to form a three-phase transformer connection, which involves high-voltage and low-voltage windings. The high-voltage windings are denoted by capital letters A-B-C, while the low-voltage windings are labeled with lowercase letters a-b-c, representing their respective phases. This notation helps distinguish between the high and low voltage sides of the transformer.
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
216
01:26Transformers
1.1K
A device that transforms voltages from one value to another using induction is called a transformer. A transformer consists of two separate coils, or windings, wrapped around the same soft iron core. However, they are electrically insulated from each other.
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
1.1K
01:26Per-Unit Sequence Models
73
An ideal Y-Y transformer, grounded through neutral impedances, displays per-unit sequence networks akin to those of a single-phase ideal transformer when subjected to balanced positive- or negative-sequence currents. These currents do not produce neutral currents, and their associated voltage drops.
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
73
01:26The Ideal Transformer
366
In single-phase two-winding transformers, two windings are coiled around a magnetic core characterized by cross-sectional area A and magnetic permeability μ. A phasor current i1 enters the left winding while i2 exits the right winding, establishing the fundamental working of the transformer through electromagnetic principles.
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
366