Stochastic Conjugate Gradient Algorithm With Variance Reduction.

Area of Science:

• Numerical Analysis
• Optimization Algorithms
• Machine Learning

Background:

• Conjugate Gradient (CG) methods are fundamental for solving linear equations and nonlinear optimization.
• Existing CG methods face challenges with large-scale and complex datasets.

Purpose of the Study:

• To develop a novel stochastic CG algorithm with enhanced variance reduction capabilities.
• To analyze the convergence properties and practical performance of the proposed algorithm.

Main Methods:

• Development of a new stochastic Conjugate Gradient algorithm incorporating variance reduction (CGVR).
• Theoretical analysis proving linear convergence for strongly convex and smooth functions using the Fletcher and Reeves method.
• Experimental validation on four learning models and six large-scale datasets.

Main Results:

• The CGVR algorithm achieves linear convergence for strongly convex and smooth functions.
• Experimental results show faster convergence compared to existing methods across diverse learning models.
• Comparable performance to LIBLINEAR on L2-regularized L2-loss problems with superior computational efficiency.

Conclusions:

• The proposed stochastic CG algorithm with variance reduction offers significant improvements in convergence speed and computational efficiency.
• CGVR is a promising alternative for solving large-scale optimization problems in machine learning.
• The algorithm's effectiveness extends to convex, nonconvex, and nonsmooth scenarios.