Stochastic Recursive Gradient Support Pursuit and Its Sparse Representation Applications.

Area of Science:

• Optimization Algorithms
• Signal Processing
• Machine Learning

Background:

• Sparse representation with L0-norm constraint is crucial in signal processing.
• Existing algorithms like SG-HT and SVRGHT have high per-iteration complexity due to frequent hard thresholding.
• This complexity hinders convergence, especially in high-dimensional scenarios.

Purpose of the Study:

• To develop a novel algorithm with reduced computational complexity for sparse representation problems.
• To improve convergence rates compared to existing stochastic hard thresholding methods.
• To validate the algorithm's effectiveness on large-scale synthetic and real-world data.

Main Methods:

• Introduction of the stochastic recursive gradient support pursuit (SRGSP) algorithm.
• SRGSP requires only one hard thresholding operation per outer iteration.
• Convergence analysis demonstrating a linear convergence rate for SRGSP.

Main Results:

• SRGSP exhibits significantly lower computational complexity than SG-HT and SVRGHT.
• The algorithm achieves a linear convergence rate.
• Experimental results show SRGSP outperforms state-of-the-art methods on synthetic and real-world datasets.

Conclusions:

• SRGSP offers a more efficient approach to solving sparse representation problems.
• The algorithm demonstrates superior performance in image denoising and face recognition.
• SRGSP provides a promising alternative for various sparse representation learning optimization tasks.