Multi-scale Fisher's independence test for multivariate dependence.

Area of Science:

• Statistics
• Machine Learning
• Data Mining

Background:

• Identifying dependencies in multivariate data is crucial for many applications.
• Existing nonparametric independence tests are computationally intensive, scaling quadratically with sample size.
• Resampling methods to assess statistical significance further increase computational burden.

Purpose of the Study:

• To introduce a scalable, resampling-free method for testing independence between random vectors.
• To address computational challenges posed by large sample sizes and high dimensionality.
• To provide a method that not only tests independence but also reveals the nature of dependencies.

Main Methods:

• A novel approach transforming independence testing into a multiple testing problem.
• Sequential coarse-to-fine discretization of the sample space into 2x2 contingency tables.
• A coarse-to-fine sequential adaptive procedure to handle increasing dimensionality by exploiting spatial dependency structures.

Main Results:

• The proposed method achieves almost linear complexity with respect to sample size.
• Guaranteed inferential validity and strong control of the testing level without resampling or asymptotic approximations.
• Demonstrated substantial computational advantages and robust statistical power compared to existing methods.

Conclusions:

• The new method offers a computationally efficient and statistically valid alternative for independence testing in large, high-dimensional datasets.
• Its divide-and-conquer nature facilitates learning the underlying dependency structures.
• Successfully applied to analyze flow cytometry data, showcasing practical utility.