Distribution of singular values in large sample cross-covariance matrices

Area of Science:

• Statistics
• Data Science
• Machine Learning

Background:

• Estimating cross-covariance for high-dimensional datasets (N > T) is challenging due to large sampling fluctuations.
• Existing methods like whitening fail when data dimensionality exceeds the number of samples.

Purpose of the Study:

• To derive the probability distribution of singular values for empirical cross-covariances of high-dimensional datasets.
• To extend the Marchenko-Pastur theorem to analyze cross-covariance matrices.
• To enable signal detection in scenarios where traditional methods are inadequate.

Main Methods:

• Analyzing uncorrelated Gaussian i.i.d. matrices X and Y with dimensions T×N_X and T×N_Y.
• Deriving the probability distribution of singular values of XᵀY in various parameter regimes.
• Investigating limiting cases of the derived distributions.

Main Results:

• The probability distribution of singular values for empirical cross-covariances is derived.
• This extends the Marchenko-Pastur result for sample covariance matrices.
• The derived distributions allow for signal detection even when N > T.

Conclusions:

• The new method provides a robust framework for analyzing cross-correlations in high-dimensional data.
• It offers a way to establish statistical significance for cross-correlations, even in challenging N > T scenarios.
• This research has broad implications for various data science applications requiring robust correlation analysis.