Adaptive Niching-Based Gradient-Accelerated Differential Evolution for High-Dimensional Nonconvex Optimization

Area of Science:

• Optimization
• Machine Learning
• Deep Learning

Background:

• Nonconvex optimization is challenging for training deep neural networks (DNNs), leading to poor generalization, especially with limited data.
• High-dimensional nonconvex optimization faces difficulties in balancing global exploration and local exploitation, and in establishing convergence guarantees, particularly with sparse individuals under nonsmooth regularizations.

Purpose of the Study:

• To introduce an adaptive niching-based gradient-accelerated differential evolution (DE) algorithm (AdaptiveGDE) to address limitations in nonconvex optimization for DNN training.
• To improve the balance between global exploration and local exploitation in optimization algorithms.
• To provide convergence guarantees for nonconvex optimization problems.

Main Methods:

• Developed AdaptiveGDE, a novel differential evolution algorithm incorporating a two-step mutation operator that decouples differential mutation and gradient descent.
• Implemented an adaptive niching strategy to dynamically adjust subpopulations based on similarity and iteration progress.
• Provided convergence guarantees under relaxed smoothness assumptions and approximate $\ell _{1}$ regularization.

Main Results:

• AdaptiveGDE demonstrated robust global exploration on complex multimodal functions and strong local exploitation on convex problems.
• The algorithm significantly improved test accuracy and reduced loss in deep neural network training, particularly in limited data scenarios.
• Achieved convergence guarantees in expectation to a near-optimal solution within $\mathcal {O}(1/\epsilon ^{4})$ iterations.

Conclusions:

• AdaptiveGDE effectively addresses key challenges in high-dimensional nonconvex optimization for deep learning.
• The proposed algorithm enhances both exploration and exploitation, leading to improved DNN performance and generalization, especially under data scarcity.
• AdaptiveGDE offers a promising approach for robust and efficient training of deep neural networks.