Weak signals in high-dimension regression: detection, estimation and prediction.

Area of Science:

• Statistics
• Econometrics
• Machine Learning

Background:

• Traditional regularization methods like Lasso, group Lasso, and SCAD prioritize strong signals, potentially leading to biased predictions when weak signals are numerous.
• Ignoring weak signals can compromise the accuracy of statistical models, especially in complex datasets.

Purpose of the Study:

• To develop a novel statistical approach that incorporates weak signals into variable selection, estimation, and prediction.
• To improve the performance of predictive models by accounting for both strong and weak effects.

Main Methods:

• A two-stage procedure involving covariance-insured screening for weak signal detection.
• Post-selection estimation using a shrinkage estimator to jointly estimate selected strong and weak signals.
• The proposed method is termed the covariance-insured screening based post-selection shrinkage estimator.

Main Results:

• The proposed method demonstrates improved estimation and prediction performance in simulation studies.
• Asymptotic properties of the new estimator have been theoretically established.
• The method was successfully applied to predict annual gross domestic product (GDP) rates.

Conclusions:

• Incorporating weak signals significantly enhances statistical estimation and prediction accuracy.
• The covariance-insured screening based post-selection shrinkage estimator offers a robust alternative to traditional methods.
• This approach has practical applications in economic forecasting and other fields requiring complex data analysis.