Unifying and revisiting Sharpness-Aware Minimization with noise-injected micro-batch scheduler for efficiency...

Area of Science:

• Machine Learning
• Deep Learning Optimization

Background:

• Sharpness-Aware Minimization (SAM) enhances model generalization by promoting flatter minima in the loss landscape.
• Standard SAM incurs a 2x computational overhead due to its two-step gradient calculation process.
• Existing efficiency improvements involve micro-batching or optimizer switching, with micro-batching approximating SAM's perturbation step.

Purpose of the Study:

• To investigate the impact of micro-batch approximated perturbations on accuracy and efficiency in SAM.
• To propose a novel method to alleviate accuracy degradation caused by small micro-batches in SAM.
• To enhance the efficiency of SAM while maintaining or improving model generalization performance.

Main Methods:

• Introduced a micro-batch scheduler to unify existing efficiency strategies for SAM.
• Revisited the role of micro-batch approximated perturbation, identifying accuracy degradation with too-small micro-batches.
• Proposed injecting random noise into the micro-batch approximated gradient during SAM's first ascent step to improve flatness.

Main Results:

• Empirically observed that small micro-batches can lead to sharper loss landscapes and reduced accuracy.
• Noise injection in the ascent step was visualized to encourage convergence to flatter regions.
• The proposed method achieved competitive accuracy with higher efficiency compared to several efficient SAM variants on CIFAR-10 and ImageNet-1K datasets.

Conclusions:

• The proposed noise injection technique effectively mitigates accuracy loss associated with micro-batch approximations in SAM.
• This approach offers a more efficient yet accurate alternative to standard SAM and other efficient variants.
• The findings provide valuable insights into optimizing deep learning models for better generalization and computational performance.