Foundational Principles for Large-Scale Inference: Illustrations Through Correlation Mining.

Area of Science:

• Statistical inference
• Machine Learning
• Data Science

Background:

• Large-scale data applications (e.g., genomics, connectomics) often face a
• variable-rich but sample-starved
• regime (n << p).
• Existing research often focuses on computational complexity, with less attention paid to sample complexity, especially when sample size (n) is fixed and variable dimension (p) grows unboundedly.

Purpose of the Study:

• To develop a unified statistical framework for quantifying sample complexity in high-dimensional inference.
• To address the gap in understanding reliable inference when dealing with a fixed sample size and an ever-increasing number of variables.
• To provide a generalizable approach applicable to various inferential tasks, including correlation mining.

Main Methods:

• Development of a unified statistical framework to explicitly quantify sample complexity.
• Categorization of sampling regimes: classical asymptotic, mixed asymptotic, and purely high-dimensional asymptotic.
• Application and illustration of the framework to correlation mining, analyzing pairwise and partial correlations.

Main Results:

• The framework explicitly quantifies sample complexity across different sampling regimes.
• The purely high-dimensional asymptotic regime is identified as most relevant for exascale data dimensions.
• Demonstration of varying correlation mining regimes based on high-dimensional learning rates and sample complexity for structured covariance models.

Conclusions:

• The developed framework provides a method to determine when reliable inference can be drawn in "Big Data" contexts.
• Understanding sample complexity is critical for effective inference in high-dimensional, sample-starved datasets.
• The framework offers a unifying perspective for correlation mining and has implications for general large-scale inference.