Embracing the Blessing of Dimensionality in Factor Models

Area of Science:

• Statistics
• High-Dimensional Data Analysis
• Factor Modeling

Background:

• Factor modeling is crucial for understanding dependencies in high-dimensional data.
• Current methods often underutilize available data for covariance matrix estimation.
• The 'blessing of dimensionality' remains underexplored in this context.

Purpose of the Study:

• To investigate whether incorporating data beyond targeted variables enhances covariance matrix estimation.
• To quantify the statistical gains (Fisher information, convergence rate) from using more data.
• To develop computationally efficient methods for full-data analysis.

Main Methods:

• Theoretical analysis providing sufficient conditions for improved estimation.
• Quantification of statistical gains using Fisher information and convergence rates.
• Development and analysis of a divide-and-conquer algorithm for computational efficiency.

Main Results:

• Using additional variables demonstrably improves covariance matrix estimation accuracy.
• An oracle-like result is achievable with sufficient data.
• The proposed divide-and-conquer algorithm matches the statistical accuracy of pooled analysis while reducing computational load.

Conclusions:

• Advocating for the use of all available data in high-dimensional factor models for superior covariance matrix estimation.
• The proposed algorithm effectively addresses computational challenges without sacrificing statistical performance.
• Empirical validation on microarray data confirms the benefits of the full-data approach.