BDS-Adam optimizer integrating adaptive variance rectification with semi-adaptive gradient smoothing.

Area of Science:

• Machine Learning
• Deep Learning Optimization
• Gradient-Based Optimization

Background:

• The Adam optimizer suffers from biased gradient estimation and early-stage training instability.
• These limitations hinder optimal performance in deep learning models.

Purpose of the Study:

• Introduce BDS-Adam, an enhanced Adam optimizer variant.
• Address biased gradients and improve training stability for deep learning.

Main Methods:

• Implemented a dual-path framework with nonlinear gradient mapping and semi-adaptive gradient smoothing.
• Utilized gradient fusion and adaptive second-order moment correction.
• Conducted convergence analysis under non-convex settings.

Main Results:

• BDS-Adam demonstrated improved robustness and long-term stability.
• Achieved test accuracy increases of 9.27% (CIFAR-10), 0.08% (MNIST), and 3.00% (gastric pathology dataset) over Adam.
• Empirical evaluations confirmed enhanced convergence speed and generalization.

Conclusions:

• BDS-Adam effectively overcomes Adam's limitations.
• The proposed optimizer enhances both convergence and generalization across various tasks.
• BDS-Adam offers a more stable and accurate training alternative.