On Consistency and Sparsity for Principal Components Analysis in High Dimensions

Area of Science:

• Statistics
• Machine Learning
• Data Science

Background:

• Principal Component Analysis (PCA) is a standard technique for dimensionality reduction.
• Modern datasets frequently exhibit more variables (p) than observations (n), posing challenges for traditional PCA.
• Standard PCA's consistency in estimating principal components degrades when p is comparable to or exceeds n.

Purpose of the Study:

• To investigate the limitations of standard PCA in high-dimensional settings (p >> n).
• To propose and validate a preprocessing step for enhancing PCA's performance on such datasets.
• To ensure the consistency of principal component estimation even when dimensionality is very high.

Main Methods:

• Development of a simple asymptotic model to analyze PCA consistency.
• Introduction of an initial dimensionality reduction strategy using sparse representations.
• An algorithm for selecting a subset of coordinates based on largest sample variances.

Main Results:

• Standard PCA's leading component estimation is consistent if and only if p(n)/n approaches 0.
• Applying PCA to a subset of coordinates with largest variances recovers consistency.
• This subset selection method enables consistent PCA even when p(n) is much larger than n (p(n) ⪢ n).

Conclusions:

• Initial dimensionality reduction is crucial for high-dimensional data before applying PCA.
• Employing a sparse representation basis for initial reduction is an effective strategy.
• The proposed subset selection algorithm enhances PCA's reliability in p >> n scenarios.