在大样本交叉协方差矩阵中奇数值的分布
Arabind Swain1, Sean Alexander Ridout2, Ilya Nemenman3
1Emory University, Department of Physics, Atlanta, Georgia 30322, USA.
Physical review. E
|October 21, 2025
概括
这项研究引入了一种新方法来分析高维数据集中的交叉相关性,扩展了Marchenko-Pastur定理. 这些发现使得即使数据尺寸超过样本大小,也可以进行信号检测,这对于数据科学应用至关重要.
科学领域:
- 统计 统计 统计 统计
- 数据科学数据科学数据科学
- 机器学习 机器学习
背景情况:
- 估计高维数据集 (N > T) 的交叉共变性是具有挑战性的,因为采样波动很大.
- 现有的白化等方法在数据维度超过样本数时失败.
研究的目的:
- 为了推导高维数据集的实证交叉协方差的奇数值的概率分布.
- 为了扩展马尔琴科-帕斯图定理来分析交叉协方差矩阵.
- 为了在传统方法不足的场景中实现信号检测.
主要方法:
- 分析无关联的高斯式i.i.d. 矩阵 X 和 Y 的尺寸为 T×N_X 和 T×N_Y.
- 在各种参数模式下推导 XTY 的奇数值的概率分布.
- 调查衍生分布的限制情况.
主要成果:
- 对于实证交叉协方差的奇数值的概率分布是导出的.
- 这扩展了Marchenko-Pastur结果的样本共变矩阵.
- 导出的分布允许信号检测,即使N > T.
结论:
- 这种新方法为分析高维数据中的交叉相关性提供了一个强大的框架.
- 它提供了一种方法来确定交叉相关性的统计意义,即使在具有挑战性的N > T场景中也是如此.
- 这项研究对各种数据科学应用具有广泛的影响,这些应用需要强大的相关性分析.
相关概念视频
One-Way ANOVA: Equal Sample Sizes
4.0K
One-Way ANOVA can be performed on three or more samples with equal or unequal sample sizes. When one-way ANOVA is performed on two datasets with samples of equal sizes, it can be easily observed that the computed F statistic is highly sensitive to the sample mean.
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
4.0K
Chi-square Distribution
6.5K
How does one determine if bingo numbers are evenly distributed or if some numbers occurred with a greater frequency? Or if the types of movies people preferred were different across different age groups or if a coffee machine dispensed approximately the same amount of coffee each time. These questions can be addressed by conducting a hypothesis test. One distribution that can be used to find answers to such questions is known as the chi-square distribution. The chi-square distribution has...
6.5K
One-Way ANOVA: Unequal Sample Sizes
6.6K
One-way ANOVA can be performed on three or more samples of unequal sizes. However, calculations get complicated when sample sizes are not always the same. So, while performing ANOVA with unequal samples size, the following equation is used:
6.6K
Distributions to Estimate Population Parameter
5.0K
The accurate values of population parameters such as population proportion, population mean, and population standard deviation (or variance) are usually unknown. These are fixed values that can only be estimated from the data collected from the samples. The estimates of each of these parameters are sample proportion, the sample mean, and sample standard deviation (or variance). To obtain the values of these sample statistics, data are required that have particular distribution and central...
5.0K
Extraction: Partition and Distribution Coefficients
4.6K
The distribution law or Nernst's distribution law is the law that governs the distribution of a solute between two immiscible solvents. This law, also known as the partition law, states that if a solute is added to the mixture of two immiscible solvents at a constant temperature, the solute is distributed between the two solvents in such a way that the ratio of solute concentrations in the solvents remains constant at equilibrium.
For extracting a solute from an aqueous phase into an...
For extracting a solute from an aqueous phase into an...
4.6K
Wilcoxon Signed-Ranks Test for Median of Single Population
432
The Wilcoxon signed-rank test for the median of a single population is a nonparametric test used to evaluate whether the median of a population differs from a specified value. Unlike parametric tests, it does not require data to follow a normal distribution, making it suitable for non-normal or small samples. The test begins by calculating the difference (d) between each observation and the hypothesized median. The absolute values of these differences are ranked in ascending order, with ties...
432


