对卷积神经网络进行有效的损失景观重塑
IEEE transactions on neural networks and learning systems
|October 1, 2024
概括
这项研究重塑了深度学习损失格局,以找到平面最小值,提高了概括性,而不会影响培训效率或稳定性. 这种新的方法在各种任务中提供了显著的性能提升,计算开销最小.
科学领域:
- 深度学习 (Deep Learning) 是一种深度学习.
- 机器学习优化优化
- 计算神经科学是一种神经科学.
背景情况:
- 在平面损失最小值和深度学习模型中改进的概括之间存在正相关性.
- 目前寻找平面最小值的方法通常涉及高计算成本或与培训稳定性和融合的权衡.
研究的目的:
- 提出一种用于重塑损失景观的新方法,以引导优化器向平坦地区指导.
- 为了实现更好的概括性能,计算成本可以忽略不计,而不会影响培训动态.
主要方法:
- 开发基于随机优化的理论见解的非线性,损失依赖的重塑函数.
- 分析低损耗景观的化和高和超低损耗景观的平整的影响.
- 在图像分类,对抗性强度和自然语言处理 (NLP) 任务上实施和评估重塑函数.
主要成果:
- 微妙设计的重塑功能有效地引导优化器到平面最小值,提高了概括性能.
- 提出的方法稳定了培训,促进了优化,并保持了计算效率.
- 在各种深度学习应用中观察到普遍化的显著改进.
结论:
- 重塑损失格局提供了一个计算上便宜且有效的策略,用于改进深度神经网络概括.
- 这种方法为训练深度神经网络提供了新的视角,平衡性能与效率.
- 这些发现对深度学习优化和模型培训领域有广泛的影响.
相关概念视频
Reducing Line Loss
149
In a three-phase circuit, line loss is an indicator of energy dissipated as heat due to the resistance of transmission lines. To address this, incorporating transformers into the system—a step-up transformer at the source and a step-down transformer at the load—is a strategic solution. Two three-phase transformers are introduced to improve this.
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
149
Boundary Conditions: Lossless Lines
88
Consider a single-phase, two-wire, lossless transmission line terminated by an impedance at the receiving end and a source with Thevenin voltage and impedance at the sending end. The line, with length, has a surge impedance and wave velocity determined by the line's inductance and capacitance.
At the receiving end, the boundary condition states that the voltage equals the product of the receiving-end impedance and current. This relationship is expressed as a function of the incident and...
At the receiving end, the boundary condition states that the voltage equals the product of the receiving-end impedance and current. This relationship is expressed as a function of the incident and...
88
Energy Losses in Transformers
841
In an ideal transformer, it is assumed that there are no energy losses, and, hence, all the power at the primary winding is transferred to the secondary winding. However, in reality, the transformers always have some energy losses, and, hence, the output power obtained at the secondary winding is less than the input power at the primary winding due to energy losses.
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
841
Lossy Lines and Overvoltages
87
Transmission-line series resistance and shunt conductance cause three primary effects: attenuation, distortion, and power losses.
Attenuation
When constant series resistance and shunt conductance are present, voltage and current equations are modified. The propagation constant indicates that voltage and current waves consist of both forward and backward traveling components. These waves attenuate as they propagate, with the attenuation factor related to the resistance and conductance. In a...
Attenuation
When constant series resistance and shunt conductance are present, voltage and current equations are modified. The propagation constant indicates that voltage and current waves consist of both forward and backward traveling components. These waves attenuate as they propagate, with the attenuation factor related to the resistance and conductance. In a...
87
Convolution: Math, Graphics, and Discrete Signals
234
In any LTI (Linear Time-Invariant) system, the convolution of two signals is denoted using a convolution operator, assuming all initial conditions are zero. The convolution integral can be divided into two parts: the zero-input or natural response and the zero-state or forced response, with t0 indicating the initial time.
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
234
Residuals and Least-Squares Property
7.3K
The vertical distance between the actual value of y and the estimated value of y. In other words, it measures the vertical distance between the actual data point and the predicted point on the line
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
7.3K


