Adaptive Niching-Based Gradient-Accelerated Differential Evolution for High-Dimensional Nonconvex Optimization | JoVE Visualize

Area of Science:

Optimization
Machine Learning
Deep Learning

Background:

Nonconvex optimization is challenging for training deep neural networks (DNNs), leading to poor generalization, especially with limited data.
High-dimensional nonconvex optimization faces difficulties in balancing global exploration and local exploitation, and in establishing convergence guarantees, particularly with sparse individuals under nonsmooth regularizations.

Purpose of the Study:

To introduce an adaptive niching-based gradient-accelerated differential evolution (DE) algorithm (AdaptiveGDE) to address limitations in nonconvex optimization for DNN training.
To improve the balance between global exploration and local exploitation in optimization algorithms.
To provide convergence guarantees for nonconvex optimization problems.

Main Methods:

Developed AdaptiveGDE, a novel differential evolution algorithm incorporating a two-step mutation operator that decouples differential mutation and gradient descent.
Implemented an adaptive niching strategy to dynamically adjust subpopulations based on similarity and iteration progress.
Provided convergence guarantees under relaxed smoothness assumptions and approximate $\ell _{1}$ regularization.

Main Results:

AdaptiveGDE demonstrated robust global exploration on complex multimodal functions and strong local exploitation on convex problems.
The algorithm significantly improved test accuracy and reduced loss in deep neural network training, particularly in limited data scenarios.
Achieved convergence guarantees in expectation to a near-optimal solution within $\mathcal {O}(1/\epsilon ^{4})$ iterations.

Conclusions:

AdaptiveGDE effectively addresses key challenges in high-dimensional nonconvex optimization for deep learning.
The proposed algorithm enhances both exploration and exploitation, leading to improved DNN performance and generalization, especially under data scarcity.
AdaptiveGDE offers a promising approach for robust and efficient training of deep neural networks.