Making Trillion Correlations Feasible in Feature Grouping and Selection

Area of Science:

• Machine Learning
• Data Mining
• Computational Statistics

Background:

• Modern databases face challenges with
• Big Dimensionality
• due to computationally intensive pairwise feature correlation calculations.
• Existing methods struggle with datasets containing millions of features.
• This computational burden hinders effective analysis and model training.

Purpose of the Study:

• To address the computational challenges of analyzing big dimensional data.
• To develop a novel learning approach for efficient identification of informative and correlated feature groups.
• To reduce the complexity of feature selection in high-dimensional datasets.

Main Methods:

• Developed a novel learning approach exploiting sparse correlations in big dimensional data.
• Incorporated linear and nonlinear correlation measures as constraints within the learning model.
• Implemented an embedded feature selection strategy with V-fold cross-validation for robust feature identification.

Main Results:

• Identified that a small subset of feature pairs significantly contributes to underlying interactions.
• Demonstrated a complexity reduction from O(m^2n) to O(mlogm + K_a mn).
• Achieved notable speedups in one-class learning on big dimensional data.
• Empirical studies on datasets up to 30 million dimensions confirmed the approach's efficacy.

Conclusions:

• The proposed approach efficiently identifies informative and correlated feature groups in big dimensional data.
• The method offers significant computational advantages over traditional correlation-based techniques.
• The embedded feature selection strategy ensures robust and stable feature identification.