统一和重新审视度意识最小化与噪音注入微批次调度器,以提高效率
Zheng Wei1, Xingjun Zhang1, Zhendong Tan1
1School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an 710049, China.
概括
敏度意识最小化 (SAM) 提高了概括性,但在计算上是昂贵的. 本研究介绍了一种噪音注入方法,以提高SAM的效率和准确性,通过更快的培训实现竞争性结果.
科学领域:
- 机器学习 机器学习
- 深度学习优化优化
背景情况:
- 敏度意识最小化 (SAM) 通过在损失景观中促进更平坦的最小值来增强模型的概括性.
- 标准SAM由于其两步梯度计算过程而导致2倍的计算开销.
- 现有的效率改进涉及微分组或优化器切换,微分组接近SAM的扰动步骤.
研究的目的:
- 调查微批量近似扰动对SAM准确性和效率的影响.
- 提出一种新的方法来缓解SAM中小微批量引起的精度退化.
- 为了提高SAM的效率,同时保持或提高模型通用化性能.
主要方法:
- 引入了微批量调度器,以统一SAM的现有效率策略.
- 重新审视了微批量近似扰动的作用,确定了微批量过小的准确性退化.
- 建议在SAM的第一个上升阶段将随机噪声注入微批量近似梯度中,以改善平坦度.
主要成果:
- 经验观察到,小微批量可以导致更清晰的损失景观和降低精度.
- 在上升阶段的噪音注入被可视化,以鼓励向更平坦的区域汇聚.
- 与CIFAR-10和ImageNet-1K数据集上的几个高效的SAM变体相比,拟议的方法实现了具有竞争力的准确性和更高的效率.
结论:
- 拟议的噪声注入技术有效地减轻了与SAM中微批量近似相关的准确性损失.
- 这种方法为标准SAM和其他高效变体提供了更高效但更准确的替代方案.
- 这些发现为优化深度学习模型提供了有价值的见解,以获得更好的概括和计算性能.
相关概念视频
Reducing Line Loss
In a three-phase circuit, line loss is an indicator of energy dissipated as heat due to the resistance of transmission lines. To address this, incorporating transformers into the system—a step-up transformer at the source and a step-down transformer at the load—is a strategic solution. Two three-phase transformers are introduced to improve this.
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss in...
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss in...
Downsampling
When considering a sampled sequence with zero values between sampling instants, one can replace it by taking every N-th value of the sequence. At these integer multiples of N, the original and sampled sequences coincide. This process, known as decimation, involves extracting every N-th sample from a sequence, thereby creating a more efficient sequence.
The Fourier transform of the decimated sequence reveals a combination of scaled and shifted versions of the original spectrum. This...
The Fourier transform of the decimated sequence reveals a combination of scaled and shifted versions of the original spectrum. This...
Upsampling
Managing signal sampling rates is essential in digital signal processing to maintain signal integrity. A decimated signal, characterized by a reduced frequency range due to its lower sampling rate, can be upsampled by inserting zeros between each sample. This upsampling process expands the original spectrum and introduces repeated spectral replicas at intervals dictated by the new Nyquist frequency. To refine this zero-inserted sequence, it is passed through a lowpass filter with a cutoff...
Maximizing the Directional Derivative
The directional derivative is a central concept in multivariable calculus that describes how a function changes at a given point when moving in a specified direction. This direction is represented by a unit vector, ensuring that only the orientation influences the rate of change. By varying the direction, different rates of change can be observed, demonstrating that the directional derivative depends strongly on the chosen direction.The directional derivative is computed using the gradient...
Methods of Medium Optimization
Optimizing growth media enhances microbial proliferation and maximizes product yield. Statistical experimental design methodologies provide structured and reproducible approaches, offering progressively higher levels of robustness and efficiency.The One-Factor-at-a-Time (OFAT) MethodThe One-Factor-at-a-Time (OFAT) method involves adjusting a single variable while keeping all others constant. However, it cannot detect interactions between variables, often leading to suboptimal outcomes when...


