Estimating the number of principal components via Split-Half Eigenvector Matching (SHEM)

Area of Science:

• Statistics
• Data Science
• Machine Learning

Background:

• Principal Component Analysis (PCA) is crucial for dimension reduction.
• Existing methods for selecting the number of principal components may be suboptimal.
• Accurate component selection is vital for reliable data analysis.

Purpose of the Study:

• To present an alternative procedure for estimating the optimal number of principal components.
• To recover the true number of independent vectors underlying data generation.
• To validate a novel approach using data splitting and similarity analysis.

Main Methods:

• Repeatedly splitting data into random halves.
• Comparing eigenvectors derived from each data half.
• Analyzing the split between high and low similarity scores to estimate component count.

Main Results:

• The proposed method demonstrates a proof of principle for estimating principal components.
• Similarity analysis across data splits provides a useful estimation approach.
• The technique is applicable to dimension reduction and similar modeling problems.

Conclusions:

• Data splitting and similarity comparison offer a viable strategy for determining the number of principal components.
• This method enhances the accuracy of dimension reduction techniques.
• The approach holds promise for improving various statistical modeling applications.