Accelerated Stochastic Conjugate Gradient for a class of convex optimization.

Area of Science:

• Optimization Theory
• Machine Learning
• Numerical Analysis

Background:

• Conjugate gradient methods are fundamental for large-scale unconstrained optimization.
• Stochastic optimization methods are crucial for handling large datasets in machine learning.

Purpose of the Study:

• To introduce a novel Accelerated Stochastic Conjugate Gradient (ASCG) algorithm.
• To address challenges in convex empirical risk minimization problems.
• To improve convergence speed and stability in stochastic optimization.

Main Methods:

• Developed ASCG algorithm integrating a variance-reduced gradient estimator.
• Incorporated a novel acceleration mechanism using a step-size deflation factor.
• Performed rigorous theoretical analysis for convergence rates.

Main Results:

• ASCG achieves an expected linear convergence rate under strong convexity.
• Demonstrated superior reduction in function values compared to non-accelerated methods.
• Numerical experiments show ASCG outperforms state-of-the-art methods on benchmark datasets.

Conclusions:

• ASCG offers enhanced stability and faster practical convergence for stochastic optimization.
• The algorithm is particularly effective for convex empirical risk minimization.
• ASCG represents a significant advancement in large-scale optimization techniques.