A Stochastic Quasi-Newton Method for Large-Scale Nonconvex Optimization With Applications

Area of Science:

• Optimization Methods
• Machine Learning Algorithms
• Numerical Analysis

Background:

• Stochastic Broyden-Fletcher-Goldfarb-Shanno (BFGS) methods face challenges with Hessian update positive definiteness and ill-conditioning in nonconvex problems.
• Existing methods require robust strategies to maintain numerical stability and computational efficiency.

Purpose of the Study:

• To propose a novel stochastic damped and regularized BFGS method to address Hessian update issues in nonconvex optimization.
• To improve the robustness and reduce the computational cost of stochastic limited-memory BFGS (LBFGS) updates.

Main Methods:

• Introduced a regularized strategy to prevent the BFGS matrix from becoming singular.
• Incorporated a new damped parameter to ensure positivity of correction pair products.
• Updated curvature information using averaged iterates at spaced intervals to reduce computational cost and enhance robustness.

Main Results:

• The proposed method demonstrated superior performance compared to the stochastic damped LBFGS (SdLBFGS) method in numerical experiments.
• Showcased particular effectiveness for problems with small sample sizes, mitigating ill-conditioned issues.
• Exhibited greater robustness to variations in batch size and memory size than the SdLBFGS method.

Conclusions:

• The novel stochastic damped and regularized BFGS method effectively addresses positive definiteness and ill-conditioning in Hessian updates for nonconvex problems.
• The method offers improved performance, robustness, and computational efficiency, particularly for small sample size and ill-conditioned scenarios in machine learning applications.