ED-SAM: Sharpness-aware minimization with energy-adjusted perturbations and direction-corrected updates.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Neural Networks

Background:

• Sharpness-Aware Minimization (SAM) is a technique to improve deep learning model generalization.
• Existing SAM methods may use suboptimal perturbation directions and fail to capture true descent directions, hindering performance.
• Addressing these limitations is crucial for advancing deep learning optimization.

Purpose of the Study:

• To introduce ED-SAM, a novel algorithm enhancing SAM's generalization and robustness.
• To develop an energy-adjusted perturbation mechanism and a direction-corrected parameter update strategy.
• To theoretically analyze ED-SAM's convergence and empirically validate its superiority.

Main Methods:

• ED-SAM employs an energy-adjusted adversarial perturbation step, modeling gradient energy using the second moment of gradients.
• It incorporates a direction-corrected parameter update strategy, combining clean and perturbed gradients.
• Theoretical convergence analysis and experiments on diverse datasets and architectures were conducted.

Main Results:

• ED-SAM effectively suppresses high-energy gradient components and adapts perturbations to gradient direction and energy.
• The direction-corrected update strategy allows optimization towards the true descent direction while maintaining flatness.
• Experiments demonstrate ED-SAM's superior generalization and robustness compared to existing SAM variants.

Conclusions:

• ED-SAM offers a significant advancement in sharpness-aware minimization techniques.
• The proposed methods for perturbation and parameter updates lead to improved generalization and robustness in deep neural networks.
• ED-SAM represents a promising direction for optimizing deep learning models.