Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer.

Area of Science:

• Machine Learning
• Deep Learning Optimization

Background:

• Deep neural networks (DNNs) struggle with generalization due to complex loss landscapes.
• Sharpness-Aware Minimization (SAM) smooths loss landscapes but incurs high computational costs by perturbing all weights.

Purpose of the Study:

• To introduce Sparse SAM (SSAM), an efficient training scheme that reduces SAM's computational overhead.
• To investigate sparse perturbation strategies for improved DNN training efficiency and generalization.

Main Methods:

• SSAM employs a binary mask for sparse weight perturbation.
• Sparse masks are derived using Fisher information and dynamic sparse training.
• Investigated various mask patterns (unstructured, structured, N:M) and perturbation implementations (explicit, implicit).

Main Results:

• SSAM achieves convergence rates comparable to SAM ($O(\log T/\sqrt{T})$).
• Experiments on CIFAR and ImageNet-1K demonstrate SSAM's superior efficiency over SAM.
• Performance is maintained or enhanced with up to 50% sparsity.

Conclusions:

• SSAM offers an effective and computationally efficient alternative to SAM for DNN training.
• Sparse perturbation is a viable strategy for optimizing DNNs without sacrificing performance.