SSR1M: A stochastic SR1 method with momentum acceleration for non-Convex optimization

Area of Science:

• Machine Learning
• Optimization Algorithms
• Computational Science

Background:

• Matrix-based second-order optimization methods show strong performance but face scalability challenges due to high computational and storage costs.
• Large-scale machine learning problems necessitate more efficient optimization techniques that balance performance with resource constraints.

Purpose of the Study:

• To introduce a novel optimization algorithm, stochastic symmetric rank-one with momentum (SSR1M), designed to overcome the limitations of traditional matrix-based methods.
• To enhance the practicality and performance of second-order optimization through vector-level operations and incorporated techniques like momentum and averaging.

Main Methods:

• Developed SSR1M by replacing batch gradients with stochastic gradients and simplifying correction terms into hyperparameters.
• Integrated momentum, exponential moving averaging, and normalization techniques to boost algorithmic performance.
• Conducted a detailed convergence analysis of SSR1M in non-convex settings.

Main Results:

• SSR1M significantly reduces storage and computational overhead by utilizing vector-level operations instead of matrix products.
• Empirical evaluations on benchmark non-convex machine learning tasks show SSR1M outperforms state-of-the-art methods.
• Demonstrated superior performance in both function value and accuracy for optimizing non-convex multinomial regression and deep neural networks.

Conclusions:

• SSR1M presents an efficient and practical solution for optimizing complex non-convex machine learning models.
• The algorithm's design makes it suitable for large-scale applications where traditional methods are computationally prohibitive.
• The study validates SSR1M's effectiveness and provides an open-source implementation for broader adoption.