通过信息瓶般的动态探索神经网络中添加式快捷方式的影响:从ResNet到变压器
Zhaoyan Lyu1, Miguel R D Rodrigues1
1Department of Electronic and Electrical Engineering, University College London, London WC1E 6BT, UK.
Entropy (Basel, Switzerland)
|November 27, 2024
概括
像ResNet和ViTs这样的深度学习模型中的附加快捷方式连接会影响培训阶段. 身份快捷键使直接压缩成为可能,绕过了早期的拟合,但仍然有效地保持了泛化.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 深度学习架构 深度学习架构
背景情况:
- 包括ResNet,Vision Transformers (ViTs) 和MLP-Mixers在内的深度学习模型取得了显著的成功.
- 附加快捷方式连接对于有效的信息流动和减轻像消失梯度这样的优化挑战至关重要.
研究的目的:
- 研究添加式快捷方式连接对深度学习模型训练的安装和压缩阶段的影响.
- 分析不同类型的快捷方式 (身份与非身份) 如何使用信息瓶方法影响概括动态.
主要方法:
- 使用信息瓶方法来分析培训动态.
- 采用Z-X和Z-Y措施作为高维数据相互信息的实际替代方案.
- 经验性地比较了具有身份快捷方式 (ISs) 和非身份快捷方式 (NISs) 的模型.
主要成果:
- 具有身份快捷方式 (IS) 的模型往往会跳过初始装配阶段,直接进入压缩阶段.
- 非身份快捷方式 (NIS) 模型遵循传统的两阶段训练过程 (装配然后压缩).
- IS模型表现出有效的压缩,并保持一般化能力,即使绕过了早期的装配阶段.
结论:
- 增量捷径连接,特别是身份捷径,显著改变深度学习模型的训练动态.
- 了解这些动态为优化现代深度学习架构提供了新的见解,以改善概括性.
相关概念视频
Energy Losses in Transformers
831
In an ideal transformer, it is assumed that there are no energy losses, and, hence, all the power at the primary winding is transferred to the secondary winding. However, in reality, the transformers always have some energy losses, and, hence, the output power obtained at the secondary winding is less than the input power at the primary winding due to energy losses.
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
831
Transformers with Off-Nominal Turns Ratios
139
In scenarios involving parallel transformers with disparate ratings, developing per-unit models requires accommodating off-nominal turns ratios. This situation arises when the selected base voltages are not proportional to the transformer’s voltage ratings. Consider a transformer where the rated voltages are related by the term a. If the chosen voltage bases satisfy a relationship involving term b, term c is defined as the ratio of these bases. This ratio is then substituted into the...
139
Convolution: Math, Graphics, and Discrete Signals
231
In any LTI (Linear Time-Invariant) system, the convolution of two signals is denoted using a convolution operator, assuming all initial conditions are zero. The convolution integral can be divided into two parts: the zero-input or natural response and the zero-state or forced response, with t0 indicating the initial time.
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
231
Convolution Properties I
137
Convolution computations can be simplified by utilizing their inherent properties.
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
137
Equivalent Circuits for Practical Transformers
396
The practical equivalent circuits of single-phase two-winding transformers exhibit significant deviations from their idealized versions due to the inherent properties of winding resistance and finite core permeability. These properties result in real and reactive power losses, affecting the transformer's performance. Understanding these deviations is crucial for designing more efficient transformers.
In a practical transformer, each winding exhibits resistance and leakage reactance. The...
In a practical transformer, each winding exhibits resistance and leakage reactance. The...
396
Neuroplasticity
298
Neuroplasticity reflects the brain's remarkable capacity to adapt and evolve, responding dynamically to learning, experiences, or injury by reorganizing its neural circuitry. This reorganization involves creating new neural connections and refining old ones through a series of biological processes that contribute to the brain's lifelong development and adaptability.
298


