A Minibatch Proximal Stochastic Recursive Gradient Algorithm Using a Trust-Region-Like Scheme and Barzilai-Borwein...

Area of Science:

• Optimization
• Machine Learning
• Convex Analysis

Background:

• Regularized empirical risk minimization (ERM) is a common problem in machine learning.
• These problems involve minimizing the sum of smooth convex functions and a nonsmooth convex function with a proximal mapping.
• Existing methods may not offer optimal convergence rates for these complex optimization landscapes.

Purpose of the Study:

• To propose a new algorithm, mSRGTR-BB (minibatch proximal stochastic recursive gradient algorithm with Barzilai-Borwein stepsizes).
• To analyze the convergence properties of mSRGTR-BB for both strongly and nonstrongly convex functions.
• To compare the performance of mSRGTR-BB against state-of-the-art methods in machine learning optimization.

Main Methods:

• Development of the mSRGTR-BB algorithm, incorporating a trust-region-like scheme for stepsize selection.
• Utilizing the Barzilai-Borwein method for automatic stepsize computation.
• Theoretical analysis to prove linear convergence in expectation for various convexity conditions.

Main Results:

• mSRGTR-BB achieves linear convergence in expectation for strongly and nonstrongly convex objective functions.
• The algorithm demonstrates a faster convergence rate compared to the minibatch proximal semistochastic gradient method (mS2GD).
• Numerical experiments show mSRGTR-BB's performance is competitive with, and sometimes exceeds, mS2GD and other modern proximal stochastic gradient methods.

Conclusions:

• mSRGTR-BB is an effective and efficient algorithm for solving regularized empirical risk minimization problems.
• The proposed method offers significant improvements in convergence speed and overall performance.
• mSRGTR-BB represents a valuable advancement for optimization in machine learning applications.