mL-BFGS: A Momentum-based L-BFGS for Distributed Large-Scale Neural Network Optimization.

Area of Science:

• Machine Learning
• Optimization Algorithms
• Deep Neural Networks

Background:

• Quasi-Newton methods, including L-BFGS, face challenges in large-scale deep neural network training due to computational costs and instability in stochastic settings.
• Existing adaptations of L-BFGS for stochastic training often introduce significant overhead, negating convergence benefits.

Purpose of the Study:

• To propose mL-BFGS, a lightweight, momentum-based L-BFGS algorithm designed for efficient large-scale distributed deep neural network optimization.
• To enhance the stability and reduce the computational burden of quasi-Newton methods in deep learning.

Main Methods:

• Developed mL-BFGS, incorporating a momentum scheme into the L-BFGS update to mitigate stochastic noise in Hessian approximations.
• Implemented block-wise Hessian approximation in mL-BFGS to distribute computational and memory costs across nodes for large-scale training.
• Provided theoretical convergence analysis for mL-BFGS in stochastic optimization scenarios.

Main Results:

• mL-BFGS demonstrated stabilized convergence during stochastic optimization by reducing noise in Hessian approximations.
• Block-wise Hessian approximation enabled efficient scaling of computation and memory for distributed training.
• Empirical results on benchmark neural models showed significant iteration-wise and wall-clock speedups compared to baseline methods like SGD and Adam.

Conclusions:

• mL-BFGS offers a promising approach for leveraging quasi-Newton methods in large-scale deep neural network training.
• The proposed algorithm effectively balances computational efficiency with convergence stability, outperforming existing methods.