Asymptotic Distribution-Free Independence Test for High Dimension Data
These videos have been matched automatically. Contact us if they are not relevant.
These videos have been matched automatically. Contact us if they are not relevant.
View abstract on PubMed
Summary
This summary is machine-generated.We introduce a new framework for independence testing in high-dimensional data. This method leverages machine learning classifiers to detect sparse dependencies, offering a powerful tool for complex datasets.
Area Of Science
- Statistics
- Machine Learning
- Data Science
Background
- Independence testing is crucial for variable selection, graphical models, and causal inference.
- High-dimensional and sparse data present significant challenges for traditional independence tests due to a lack of distributional or structural assumptions.
Purpose Of The Study
- To propose a general and robust framework for independence testing applicable to high-dimensional, complex data.
- To develop a test statistic with a universal, fixed Gaussian null distribution, independent of the data distribution.
Main Methods
- A novel framework for independence testing by fitting a classifier to distinguish joint and product distributions.
- Utilizing advanced classification algorithms from machine learning.
- Employing a sample split and fixed permutation strategy to ensure a fixed Gaussian null distribution.
Main Results
- The proposed test demonstrates advantages over existing methods in extensive simulations.
- The framework effectively handles high-dimensional and sparse data, outperforming current approaches.
- Successful application to a single-cell sequencing dataset for testing independence between measurement types.
Conclusions
- The new framework offers a powerful and flexible approach to independence testing, particularly for complex, high-dimensional data.
- The method's ability to leverage machine learning enhances its applicability in modern data analysis.
- The universal null distribution simplifies interpretation and broadens the scope of application.
These videos have been matched automatically. Contact us if they are not relevant.
These videos have been matched automatically. Contact us if they are not relevant.
Related Concept Videos
In statistics, the term independence means that one can directly obtain the probability of any event involving both variables by multiplying their individual probabilities. Tests of independence are chi-square tests involving the use of a contingency table of observed (data) values.
The test statistic for a test of independence is similar to that of a goodness-of-fit test:
where:
O = observed values
E = expected values (which should be at least 5)
A test of independence determines whether...
The test of independence is a chi-square-based test used to determine whether two variables or factors are independent or dependent. This hypothesis test is used to examine the independence of the variables. One can construct two qualitative survey questions or experiments based on the variables in a contingency table. The goal is to see if the two variables are unrelated (independent) or related (dependent). The null and alternative hypotheses for this test are:
H0: The two variables (factors)...
The goodness–of–fit test can be used to decide whether a population fits a given distribution, but it will not suffice to decide whether two populations follow the same unknown distribution. A different test, called the test for homogeneity, can be used to conclude whether two populations have the same distribution. To calculate the test statistic for a test for homogeneity, follow the same procedure as with the test of independence. The hypotheses for the test for homogeneity can...
One-Way ANOVA can be performed on three or more samples with equal or unequal sample sizes. When one-way ANOVA is performed on two datasets with samples of equal sizes, it can be easily observed that the computed F statistic is highly sensitive to the sample mean.
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
The F distribution was named after Sir Ronald Fisher, an English statistician. The F statistic is a ratio (a fraction) with two sets of degrees of freedom; one for the numerator and one for the denominator. The F distribution is derived from the Student's t distribution. The values of the F distribution are squares of the corresponding values of the t distribution. One-Way ANOVA expands the t test for comparing more than two groups. The scope of that derivation is beyond the level of this...
One-way ANOVA can be performed on three or more samples of unequal sizes. However, calculations get complicated when sample sizes are not always the same. So, while performing ANOVA with unequal samples size, the following equation is used:
In the equation, n is the sample size, ͞x is the sample mean, x̿ is the combined mean for all the observations, k is the number of samples, and s2 is the variance of the sample. It should be noted that the subscript 'i'...