信号传播的几何动态预测了变压器的可训练性
Aditya Cowsik1, Tamra Nebabu2, Xiaoliang Qi1
1Stanford University, Department of Physics, Stanford, California 94305, USA.
Physical review. E
|December 23, 2025
概括
我们提出了初始化深度变压器的新标准,以确保可训练性. 我们的方法分析信号传播和梯度动态,识别与最佳超参数设置相关的秩序混乱过渡,以最小的测试损失.
科学领域:
- 深度学习 (Deep Learning) 是一种深度学习.
- 人工智能的人工智能
- 神经网络架构 神经网络架构
背景情况:
- 深度变压器是强大的,但由于信号传播和梯度动态的问题,训练具有挑战性.
- 随机初始化通常会导致梯度爆炸或消失,阻碍有效的学习.
研究的目的:
- 在深度变压器中为超参数初始化开发具有几何意义的标准.
- 通过分析前向和后向信号传播,确保深度变压器的可训练性.
主要方法:
- 在深度,随机初始化的变压器中研究了前进信号和梯度反向传播.
- 利用动态视角,将代币向量视为通过变压器层进化的粒子.
- 导出组合平均的几何更新方程和分析的相位过渡.
主要成果:
- 在向前信号传播 (标记角度) 中确定了一个秩序-混乱相位过渡.
- 在反向传播中观察到爆炸/消失梯度之间的过渡.
- 导出相位边界和两个利亚普诺夫指数 (标记角度和梯度指数).
结论:
- 标记角度和梯度指数的同时消失为超参数初始化提供了简单的标准.
- 这些标准与最小的测试损失相关,这表明了更稳定,更有效的深度变压器训练的途径.
相关概念视频
Energy Losses in Transformers
1.3K
In an ideal transformer, it is assumed that there are no energy losses, and, hence, all the power at the primary winding is transferred to the secondary winding. However, in reality, the transformers always have some energy losses, and, hence, the output power obtained at the secondary winding is less than the input power at the primary winding due to energy losses.
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
1.3K
Transformers
1.7K
A device that transforms voltages from one value to another using induction is called a transformer. A transformer consists of two separate coils, or windings, wrapped around the same soft iron core. However, they are electrically insulated from each other.
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
1.7K
Three-Winding Transformers
648
Three identical single-phase transformers can be configured to form a three-phase transformer connection, which involves high-voltage and low-voltage windings. The high-voltage windings are denoted by capital letters A-B-C, while the low-voltage windings are labeled with lowercase letters a-b-c, representing their respective phases. This notation helps distinguish between the high and low voltage sides of the transformer.
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
648
Equivalent Circuits for Practical Transformers
1.3K
The practical equivalent circuits of single-phase two-winding transformers exhibit significant deviations from their idealized versions due to the inherent properties of winding resistance and finite core permeability. These properties result in real and reactive power losses, affecting the transformer's performance. Understanding these deviations is crucial for designing more efficient transformers.
In a practical transformer, each winding exhibits resistance and leakage reactance. The...
In a practical transformer, each winding exhibits resistance and leakage reactance. The...
1.3K
Transformers with Off-Nominal Turns Ratios
490
In scenarios involving parallel transformers with disparate ratings, developing per-unit models requires accommodating off-nominal turns ratios. This situation arises when the selected base voltages are not proportional to the transformer’s voltage ratings. Consider a transformer where the rated voltages are related by the term a. If the chosen voltage bases satisfy a relationship involving term b, term c is defined as the ratio of these bases. This ratio is then substituted into the...
490
The Ideal Transformer
1.3K
In single-phase two-winding transformers, two windings are coiled around a magnetic core characterized by cross-sectional area A and magnetic permeability μ. A phasor current i1 enters the left winding while i2 exits the right winding, establishing the fundamental working of the transformer through electromagnetic principles.
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's tangential...
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's tangential...
1.3K


