A zeroing neural dynamics based acceleration optimization approach for optimizers in deep neural networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• First-order optimizers are crucial for deep neural network (DNN) convergence.
• Existing optimizers have limitations in transferability across different application scenarios.
• Current optimizer modifications often lack generalizability.

Purpose of the Study:

• To propose a novel optimization approach integrating zeroing neural dynamics (ZND) with first-order optimizers in DNNs.
• To enhance the efficiency and effectiveness of gradient information processing.
• To provide a generic optimization method applicable to various DNN tasks.

Main Methods:

• Integration of ZND with DNN first-order optimizers via activation functions.
• Systematic mathematical derivations for ZND gradient information transformation.
• Comparative experiments on benchmark datasets (Reuters, CIFAR, MNIST) with diverse loss functions and network architectures.

Main Results:

• The proposed ZND-based approach accelerates gradient solving.
• Demonstrated reduction in loss and increase in accuracy compared to standard optimizers.
• Effective performance across different datasets, loss functions, and network frameworks.

Conclusions:

• The ZND-based optimization approach offers a significant improvement for first-order optimizers in DNNs.
• This method provides a transferable and generic solution for diverse machine learning applications.
• The integration of ZND represents a novel contribution to the field of DNN optimization.