噪声平衡和随机梯度下降的静止分布
Liu Ziyin1,2, Hongchao Li3, Masahito Ueda3,4,5
1Massachusetts Institute of Technology, Research Laboratory of Electronics, 77 Massachusetts Ave, Cambridge, Massachusetts 02139, USA.
Physical review. E
|August 1, 2025
概括
通过分析损失函数对称性,可以更好地理解神经网络的随机梯度下降 (SGD) 训练. 这些对称性揭示了SGD噪音如何调节解决方案,并导致独特的深度网络现象.
科学领域:
- 机器学习 机器学习
- 深度学习理论 深度学习理论
- 优化算法 优化算法
背景情况:
- 随机梯度下降 (SGD) 对于训练神经网络至关重要.
- 对于SGD在复杂的损失场景中进行的导航,人们对其了解甚少.
- 了解SGD动态是改善深度学习模型的关键.
研究的目的:
- 阐明SGD如何导航非线性和退化的神经网络损失景观.
- 调查小批量噪声和损失函数对称性在SGD中的作用.
- 分析SGD在深层与浅层网络中的独特行为.
主要方法:
- 在损失函数对称性存在时,对SGD动态的理论分析.
- 在线网络中对随机梯度流的静止分布的导出.
- 检查诸如相位转换和破碎的ergodicity之类的现象.
主要成果:
- 在SGD中的小批量噪声在存在重新缩放对称性时,作为对噪声平衡解决方案的调节器.
- 损失函数对称性对于探测SGD行为至关重要.
- 深度线性网络中的静态分布表现出复杂的非线性现象.
结论:
- 损失函数对称性为SGD的运作提供了关键的见解.
- 深度神经网络在SGD下表现出独特的非线性现象,这些现象在浅层模型中并未见到.
- 这项工作加深了SGD的理论理解,并突出了深度学习架构的基本差异.
相关概念视频
Maxwell-Boltzmann Distribution: Problem Solving
1.7K
Individual molecules in a gas move in random directions, but a gas containing numerous molecules has a predictable distribution of molecular speeds, which is known as the Maxwell-Boltzmann distribution, f(v).
This distribution function f(v) is defined by saying that the expected number N (v1,v2) of particles with speeds between v1 and v2 is given by
This distribution function f(v) is defined by saying that the expected number N (v1,v2) of particles with speeds between v1 and v2 is given by
1.7K
Gradient and Del Operator
2.9K
In mathematics and physics, the gradient and del operator are fundamental concepts used to describe the behavior of functions and fields in space. The gradient is a mathematical operator that gives both the magnitude and direction of the maximum spatial rate of change. Consider a person standing on a mountain. The slope of the mountain at any given point is not defined unless it is quantified in a particular direction. For this reason, a "directional derivative" is defined, which is a vector...
2.9K
Bernoulli's Equation for Flow Normal to a Streamline
944
Bernoulli's equation for flow normal to a streamline explains how pressure varies across curved streamlines due to the outward centrifugal forces induced by the fluid's curvature. The pressure is higher on the inner side of the curve, near the center of curvature, and decreases outward to balance these centrifugal forces.
The pressure difference depends on the fluid's velocity and radius of curvature. The pressure variation is minimal in flows with nearly straight streamlines.
The pressure difference depends on the fluid's velocity and radius of curvature. The pressure variation is minimal in flows with nearly straight streamlines.
944
Uniform Distribution
5.2K
The uniform distribution is a continuous probability distribution of events with an equal probability of occurrence. This distribution is rectangular.
Two essential properties of this distribution are
Two essential properties of this distribution are
5.2K
Bernoulli's Equation for Flow Along a Streamline
1.1K
Bernoulli's equation relates the energy conservation in a fluid moving along a streamline. The equation applies to incompressible and inviscid fluids under steady flow. For such a flow, Newton's second law is applied to a small fluid element, which experiences forces due to pressure differences, gravity, and velocity variations. The force balance leads to the following form of Bernoulli's equation:
1.1K
Navier–Stokes Equations
743
For incompressible Newtonian fluids, where density remains constant, stresses show a linear relationship with the deformation rate, defined by normal and shear stresses. Normal stresses depend on the pressure exerted on the fluid and the rate of deformation in specific directions, which determines how fluid flows under varying pressures. Shear stresses, on the other hand, act tangentially across fluid layers. They explain how adjacent fluid layers slide relative to one another, connecting...
743


