New optimization algorithms for neural network training using operator splitting techniques

Area of Science:

• Artificial Intelligence
• Machine Learning
• Numerical Analysis

Background:

• Neural network training relies heavily on efficient optimization algorithms.
• Existing optimizers face challenges with convergence speed and local minima.
• Dynamical systems offer potential for developing novel optimization strategies.

Purpose of the Study:

• To present a new class of optimization algorithms tailored for neural network training.
• To explore the application of sequential operator splitting techniques from dynamical systems.
• To empirically evaluate the convergence properties of these novel optimizers.

Main Methods:

• Developing optimization algorithms based on sequential operator splitting of associated dynamical systems.
• Conducting numerical simulations to assess convergence rates.
• Tuning hyper-parameters to optimize performance.
• Validating convergence using accuracy and loss metrics.

Main Results:

• Demonstrated empirical convergence of the proposed iterative schemes towards a local minimum of the loss function.
• Achieved effective convergence on MNIST, MNIST-Fashion, and CIFAR-10 classification tasks.
• Showcased the impact of hyper-parameter choices on convergence behavior.

Conclusions:

• The proposed optimization algorithms, grounded in dynamical systems and operator splitting, offer a viable alternative for neural network training.
• These methods exhibit promising convergence properties and performance on standard image classification datasets.
• Further research into hyper-parameter optimization can enhance the efficiency of these novel optimizers.