Ape Optimizer: A p-Power Adaptive Filter-Based Approach for Deep Learning Optimization.

Area of Science:

• Deep Learning
• Optimization Algorithms
• Adaptive Filtering

Background:

• Current deep learning optimizers (e.g., SGD, Adam) assume Gaussian gradient noise.
• Empirical evidence suggests gradient noise often follows heavy-tailed $\alpha $-stable distributions.
• This discrepancy challenges the performance and robustness of existing optimizers.

Purpose of the Study:

• Introduce a novel deep learning optimizer, Ape.
• Address the limitations of optimizers designed for Gaussian noise.
• Improve deep learning model accuracy and training speed.

Main Methods:

• Developed the Ape optimizer inspired by the least mean p-power (LMP) algorithm.
• Integrated a p-power adjustment mechanism to manage gradient magnitudes.
• Implemented second moment estimation tailored for $\alpha $-stable distributions.

Main Results:

• Ape demonstrated improved accuracy and faster training speeds on benchmark datasets.
• The optimizer effectively mitigates the impact of heavy-tailed gradient noise.
• Comparative experiments showed Ape outperforming existing optimizers.

Conclusions:

• Ape offers a robust solution for deep learning optimization with non-Gaussian noise.
• Cross-disciplinary inspiration can advance deep learning techniques.
• The study provides a foundation for future innovations in optimization.