Sparse Learning with Stochastic Composite Optimization.

Area of Science:

• Optimization Theory
• Machine Learning
• Sparse Learning

Background:

• Stochastic Composite Optimization (SCO) algorithms aim for optimal convergence rates.
• Existing SCO methods struggle to produce sparse solutions due to regularization and online-to-batch conversion limitations.
• High-probability bounds in current SCO are often worse than expected convergence rates.

Purpose of the Study:

• To propose a novel two-phase SCO scheme for effective sparse learning.
• To address limitations in sparsity regularization and online-to-batch conversion in existing SCO algorithms.
• To improve high-probability convergence bounds for sparse learning.

Main Methods:

• Developed a two-phase Stochastic Composite Optimization scheme.
• Introduced a novel sparse online-to-batch conversion technique.
• Proposed three concrete algorithms: OptimalSL, LastSL, and AverageSL.

Main Results:

• The proposed scheme effectively delivers sparse solutions.
• Achieved improved high-probability bounds of approximately O(log(log(T)/δ)/λT).
• Theoretical analysis and experiments confirm the superiority of the new methods.

Conclusions:

• The novel two-phase SCO scheme significantly enhances sparse learning capabilities.
• The proposed methods outperform existing approaches in achieving sparse solutions and improving convergence guarantees.
• This work offers a more effective approach to stochastic composite optimization for sparse learning.