SGD的限制动力学:修改损失,相位振荡和异常扩散
Daniel Kunin1, Javier Sagastuy-Brena2, Lauren Gillespie3
1Stanford University, Stanford, CA 94305, U.S.A. kunin@stanford.edu.
Neural computation
|December 5, 2023
概括
经过随机梯度下降训练的深度神经网络在参数空间中表现出异常扩散. 这项研究揭示了优化超参数,梯度噪声和赫西矩阵如何解释这种行为.
科学领域:
- 机器学习 机器学习
- 统计物理 统计物理
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 经过随机梯度下降 (SGD) 训练的深度神经网络 (DNN) 呈现出超出融合的复杂动态.
- 在DNN参数空间中观察到异常扩散,在DNN参数空间中,行驶距离以梯度更新的功率定律为尺度.
- 推动这些长期动态的基本机制仍然不完全理解.
研究的目的:
- 阐明SGD训练的DNN中异常扩散的机械起源.
- 揭示了优化超参数,梯度噪声结构和黑西矩阵之间的复杂相互作用.
- 开发一个理论框架,解释深度学习模型的限制动态.
主要方法:
- 导出SGD的连续时间模型作为一个低压的朗格温方程,结合有限的学习速率和批量大小.
- 对线性回归模型的分析,以获得参数和速度动态的精确分析表达式.
- 应用福克-普朗克方程来识别相空间动态的关键驱动因素,包括修改损失和概率电流.
主要成果:
- 在DNN中异常扩散是由优化超参数,梯度噪声和赫森矩阵的相互作用解释的.
- 一个修改的损失函数隐含地调整速度,并且概率电流诱导相位空间振荡,驱动观察到的动态.
- 该理论的预测与在ImageNet上训练的ResNet-18模型的动态得到了验证.
结论:
- 该研究提供了基于统计物理的机制解释,以SGD训练的深度神经网络的异常限制动态.
- 了解这些动态,受批量大小,学习速度和势头等超参数的影响,对于未来的算法改进至关重要.
- 这项工作为开发新的优化策略提供了基础,这些优化策略利用这些见解来提高性能.
更多相关视频
10:44Inherent Dynamics Visualizer, an Interactive Application for Evaluating and Visualizing Outputs from a Gene Regulatory Network Inference Pipeline
Published on: December 7, 2021
2.2K
11:03An Analog Macroscopic Technique for Studying Molecular Hydrodynamic Processes in Dense Gases and Liquids
Published on: December 4, 2017
8.6K
相关概念视频
Damped Oscillations
5.7K
In the real world, oscillations seldom follow true simple harmonic motion. A system that continues its motion indefinitely without losing its amplitude is termed undamped. However, friction of some sort usually dampens the motion, so it fades away or needs more force to continue. For example, a guitar string stops oscillating a few seconds after being plucked. Similarly, one must continually push a swing to keep a child swinging on a playground.
Although friction and other non-conservative...
Although friction and other non-conservative...
5.7K
Linear Approximation in Time Domain
83
Nonlinear systems often require sophisticated approaches for accurate modeling and analysis, with state-space representation being particularly effective. This method is especially useful for systems where variables and parameters vary with time or operating conditions, such as in a simple pendulum or a translational mechanical system with nonlinear springs.
For a simple pendulum with a mass evenly distributed along its length and the center of mass located at half the pendulum's length,...
For a simple pendulum with a mass evenly distributed along its length and the center of mass located at half the pendulum's length,...
83
Types of Damping
6.5K
If the amount of damping in a system is gradually increased, the period and frequency start to become affected because damping opposes, and hence slows, the back and forth motion (the net force is smaller in both directions). If there is a very large amount of damping, the system does not even oscillate; instead, it slowly moves toward equilibrium. In brief, an overdamped system moves slowly towards equilibrium, whereas an underdamped system moves quickly to equilibrium but will oscillate about...
6.5K
Time-Domain Interpretation of PD Control
119
Proportional-Derivative (PD) control is a widely used control method in various engineering systems to enhance stability and performance. In a system with only proportional control, common issues include high maximum overshoot and oscillation, observed in both the error signal and its rate of change. This behavior can be divided into three distinct phases: initial overshoot, subsequent undershoot, and gradual stabilization.
Consider the example of control of motor torque. Initially, a positive...
Consider the example of control of motor torque. Initially, a positive...
119
Second Order systems II
113
In an underdamped second-order system, where the damping ratio ζ is between 0 and 1, a unit-step input results in a transfer function that, when transformed using the inverse Laplace method, reveals the output response. The output exhibits a damped sinusoidal oscillation, and the difference between the input and output is termed the error signal. This error signal also demonstrates damped oscillatory behavior. Eventually, as the system reaches a steady state, the error diminishes to zero.
113
Genetic Drift
39.8K
Natural selection—probably the most well-known evolutionary mechanism—increases the prevalence of traits that enhance survival and reproduction. However, evolution does not merely propagate favorable traits, nor does it always benefit populations.
39.8K
