在深度反复网络中,在插值过渡附近的玻璃动态
1University of Copenhagen, Niels Bohr Institute, KTH, Stockholm University, Nordita, Sweden and , Denmark.
Physical review. E
|June 19, 2025
概括
深度循环网络在插值过渡附近呈现临界减速,当网络宽度接近临界值时,学习时间会分离. 这种现象,以及衰老,反映了旋转玻璃模型中的行为.
科学领域:
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
- 统计物理 统计物理
背景情况:
- 深度反复网络是复杂的模型,其行为受到网络架构的影响.
- 在深度学习中,插值过渡标志着低参数化和过度参数化模式之间的界限.
- 了解学习动态对于优化模型训练和性能至关重要.
研究的目的:
- 为了研究在断过渡附近的深度循环网络的学习动态.
- 在培训过程中分析关键的减速和衰老现象.
- 为了在深度学习动态和旋转玻璃模型之间进行平行.
主要方法:
- 在巴赫合唱曲上训练深度循环网络,使用随机梯度下降.
- 分析深度-宽度边界附近的学习时间和重量波动.
- 将观察到的现象与自旋玻璃模型的预测进行比较.
主要成果:
- 随着网络宽度从过度参数化的方面接近临界值,观察到学习的临界减速.
- 通过插值过渡确定了临界宽的零损失极限.
- 标志着体重波动的衰老,在低参数化阶段表现出缩放行为,在损失极限附近崩.
结论:
- 深度循环网络学习表现出关键的减速和老化,类似于旋转玻璃模型.
- 插值过渡是管理这些学习动态的一个关键特征.
- 旋转玻璃模型捕捉了深度学习动态的重要方面,为未来的研究提供了洞察力.
相关概念视频
Reconstruction of Signal using Interpolation
360
Signal processing techniques are essential for accurately converting continuous signals to digital formats and vice versa. When a continuous signal is sampled with a period T, the resulting sampled signal exhibits replicas of the original spectrum in the frequency domain, spaced at intervals equal to the sampling frequency. To handle this sampled signal, a zero-order hold method can be applied, which creates a piecewise constant signal by retaining each sample's value until the next...
360
Phase Transitions: Sublimation and Deposition
18.1K
Some solids can transition directly into the gaseous state, bypassing the liquid state, via a process known as sublimation. At room temperature and standard pressure, a piece of dry ice (solid CO2) sublimes, appearing to gradually disappear without ever forming any liquid. Snow and ice sublimate at temperatures below the melting point of water, a slow process that may be accelerated by winds and the reduced atmospheric pressures at high altitudes. When solid iodine is warmed, the solid sublimes...
18.1K
Phase Transitions
20.4K
Whether solid, liquid, or gas, a substance's state depends on the order and arrangement of its particles (atoms, molecules, or ions). Particles in the solid pack closely together, generally in a pattern. The particles vibrate about their fixed positions but do not move or squeeze past their neighbors. In liquids, although the particles are closely spaced, they are randomly arranged. The position of the particles are not fixed—that is, they are free to move past their neighbors to...
20.4K
Propagation of Action Potentials
7.0K
The propagation of an action potential refers to the process by which a nerve impulse, or "action potential," travels along a neuron.
Neurons (nerve cells) have a resting membrane potential, with a slightly negative charge inside compared to outside. This is maintained by ion channels, such as sodium (Na+) and potassium (K+) channels, which control the flow of ions. When a stimulus, like a touch or a signal from another neuron, triggers the neuron, sodium channels open, allowing sodium ions to...
Neurons (nerve cells) have a resting membrane potential, with a slightly negative charge inside compared to outside. This is maintained by ion channels, such as sodium (Na+) and potassium (K+) channels, which control the flow of ions. When a stimulus, like a touch or a signal from another neuron, triggers the neuron, sodium channels open, allowing sodium ions to...
7.0K
Phase Transitions: Vaporization and Condensation
18.8K
The physical form of a substance changes on changing its temperature. For example, raising the temperature of a liquid causes the liquid to vaporize (convert into vapor). The process is called vaporization—a surface phenomenon. Vaporization occurs when the thermal motion of the molecules overcome the intermolecular forces, and the molecules (at the surface) escape into the gaseous state. When a liquid vaporizes in a closed container, gas molecules cannot escape. As these gas phase...
18.8K
Gradually Varying Flow
130
Gradually varying flow (GVF) in open channels describes situations where water depth changes slowly along the channel due to factors like non-uniform bed slope, channel shape variations, or obstructions. This flow type occurs when the depth adjusts gradually to balance gravitational forces, shear forces, and energy requirements, resulting in a low rate of depth change.Characteristics of Gradually Varying FlowGVF is commonly observed in natural streams, rivers, and canals, where flow depth...
130

