GCSAM: Gradient Centralized Sharpness Aware Minimization

Area of Science:

• Deep Learning
• Computer Vision
• Medical Imaging Analysis

Background:

• Deep neural networks (DNNs) require robust generalization for reliable performance on unseen data.
• Sharpness-based measures, such as Sharpness-Aware Minimization (SAM), promote generalization by finding flatter minima.
• Existing methods like SAM face challenges with computational overhead and gradient noise, limiting scalability.

Purpose of the Study:

• To introduce Gradient-Centralized Sharpness-Aware Minimization (GCSAM) as an improved optimization technique.
• To address the limitations of SAM, including computational cost and gradient sensitivity.
• To enhance the generalization performance and efficiency of deep learning models.

Main Methods:

• Proposed GCSAM, integrating Gradient Centralization (GC) with SAM.
• Normalized gradients before the ascent step to stabilize training and reduce noise.
• Evaluated GCSAM on general vision datasets (CIFAR-10, CIFAR-100) and medical imaging datasets (breast ultrasound, COVID-19 chest X-rays).

Main Results:

• GCSAM demonstrated superior generalization performance compared to SAM and the Adam optimizer.
• The proposed method showed improved computational efficiency.
• Consistent outperformance was observed across both general and medical imaging benchmarks.

Conclusions:

• GCSAM offers a more stable and efficient approach to improving deep learning generalization.
• The technique shows significant potential for enhancing the reliability of models in critical applications like medical image analysis.
• GCSAM provides a promising alternative for optimizing deep neural networks where robust performance on unseen data is crucial.