A scalable second order optimizer with an adaptive trust region for neural networks

Area of Science:

• Machine Learning
• Optimization Algorithms

Background:

• Second-order optimization methods offer improved gradient estimations but face challenges with computational cost and memory usage due to large matrix inversions.
• Existing optimizers often rely on first-order approximations, limiting convergence speed and stability in complex deep learning models.

Purpose of the Study:

• To introduce Tadam (Trust region ADAptive Moment estimation), a novel second-order optimization algorithm designed to overcome the computational and memory limitations of traditional second-order methods.
• To develop an optimizer that efficiently approximates the Fisher information matrix and employs an adaptive trust region for improved stability and performance.

Main Methods:

• Tadam approximates the loss up to the second order using a judicious approximation of the Fisher information matrix, reducing computational complexity to O(N).
• An adaptive trust region scheme is implemented to dynamically adjust the trust region size based on loss minimization progress, mitigating approximation errors and ensuring stability.
• The optimizer internally adjusts the learning rate, providing adaptive learning rate control even when a fixed learning rate is supplied.

Main Results:

• Experiments demonstrate that Tadam outperforms established optimizers like Adam, AMSGrad, Radam, and Nadam in terms of speed and stability.
• Tadam achieves comparable space and time complexity to the benchmarked optimizers while delivering superior performance.
• The proposed method successfully finds reasonable solutions efficiently and stably across various experimental setups.

Conclusions:

• Tadam presents a computationally efficient and stable second-order optimization method suitable for deep learning applications.
• The adaptive trust region and Fisher information matrix approximation strategies effectively address the practical challenges of second-order optimization.
• Tadam offers a promising alternative to existing optimizers, providing faster and more reliable convergence.