Riemannian Acceleration for Sparse PCA With Separable Structure and Second-Order Information Exploration

Sparse Principal Component Analysis (SPCA) is a powerful technique for dimensionality reduction and feature extraction in high-dimensional data, with applications spanning various fields such as computer vision, pattern recognition, and data mining. However, the computational intensity of SPCA presents a significant challenge, necessitating the development of efficient and robust algorithms. In this paper, we shed light on the SPCA problem and uncover intriguing structures that enable us to design an efficient algorithm, which we have named SPCA_ACC. Firstly, we identify a separable structure in this problem, which prompts us to draw on the Variable Projection (VP) strategy and generalize it to separable nonlinear problem in Stiefel manifold. This strategy projects out part of the parameters to obtain a reduced problems, allowing the SPCA_ACC algorithm to optimize in a lower-dimensional parameter space. Secondly, we resolve the coupling between different parameters of the SPCA problem in the optimization process on a fixed coordinate-sparsity manifold, which opens the way to the use of second-order Riemannian accelerated VP strategy. Moreover, we systematically analyze the advantages of using VP to solve the SPCA problem from a theoretical perspective, and confirm the local quadratic convergence of our algorithm. Numerical experiments on datasets of different sizes and types demonstrate that our method achieves rapid convergence and significantly reduces computational costs.