On relative convergence properties of principal component analysis algorithms

Area of Science:

• Machine Learning
• Statistics
• Numerical Analysis

Background:

• Principal Component Analysis (PCA) is a widely used dimensionality reduction technique.
• Stochastic approximation algorithms are employed for large-scale PCA.
• Understanding convergence properties is crucial for algorithm selection.

Purpose of the Study:

• To investigate and compare the convergence properties of two stochastic approximation algorithms for PCA.
• To analytically explain observed experimental results regarding algorithm performance.
• To identify conditions favoring smaller asymptotic mean square errors (AMSE) and faster convergence rates.

Main Methods:

• Utilizing the theory of stochastic approximation, including results from Fabian.
• Analyzing asymptotic mean square errors (AMSE) for both algorithms.
• Conducting experimental studies with multidimensional Gaussian data to corroborate analytical findings.
• Examining convergence rates under various conditions.

Main Results:

• Conditions for achieving smaller AMSE for each algorithm are identified.
• Conditions determining which algorithm yields a smaller AMSE are established.
• Experimental results confirm analytical findings on AMSE and convergence.
• Factors influencing faster convergence rates for one algorithm over the other are revealed.
• One algorithm, despite higher per-iteration computation, shows better AMSE and convergence for minor eigenvectors.

Conclusions:

• The study provides analytical insights into the performance of two PCA stochastic approximation algorithms.
• Algorithm selection can be optimized based on desired AMSE and convergence speed, particularly for minor eigenvectors.
• The findings offer guidance for practical applications of PCA in data analysis.