Generic Feature Selection with Short Fat Data.

Area of Science:

• Statistics
• Machine Learning
• Bioinformatics

Background:

• High-dimensional regression (p >> n) poses challenges for accurate inference.
• Traditional methods struggle with numerous explanatory variables relative to data points.
• Variable selection is crucial for reliable model estimation in such scenarios.

Purpose of the Study:

• To develop and evaluate a novel approach for variable selection in high-dimensional regression.
• To improve the estimation of regression coefficients when the number of predictors exceeds the sample size.
• To explore the impact of clustering, statistics, and penalty terms on model performance.

Main Methods:

• Grouping numerous explanatory variables (p) into blocks using clustering algorithms.
• Evaluating block statistics to represent variable groups.
• Regressing the response variable on these block statistics using a penalized error criterion.
• Examining performance across various choices of sample size (n), number of variables (p), statistics, clustering methods, penalty terms, and data types.

Main Results:

• The proposed block-based regression approach enhances coefficient estimation in high-dimensional settings.
• Optimal performance is suggested when regressing on approximately n/K statistics, where K is the number of clusters.
• Deviations from this optimum occur with highly variable block sizes and certain L-norm penalty terms (high q).

Conclusions:

• Clustering explanatory variables into blocks offers an effective strategy for variable selection in p >> n regression.
• The number of statistics used for regression should be carefully chosen, guided by the ratio of data points to clusters.
• The choice of penalty term significantly influences the effectiveness of the method.