Phase transition and higher order analysis of L regularization under dependence

Area of Science:

• Statistics
• Signal Processing
• Machine Learning

Background:

• Estimating sparse signals from noisy observations is a fundamental problem.
• The L1-regularized least squares (LQLS) method is widely used for sparse signal recovery.

Purpose of the Study:

• To analyze the asymptotic risk of LQLS for sparse signal estimation.
• To investigate the influence of predictor covariance structure on LQLS performance.

Main Methods:

• Derivation of asymptotic risk using the replica method.
• Higher-order analysis in the small-error regime.
• Explicit formula derivation for dominant terms in asymptotic risk expansion.

Main Results:

• Asymptotic risk derived for arbitrary covariance matrices, generalizing Gaussian design results.
• First dominant term of risk is independent of covariance for certain LQLS variants.
• Correlations among predictors affect phase transitions only for LASSO (a specific LQLS case).
• Explicit formulas for the second dominant term derived to study covariance influence.

Conclusions:

• The study provides a theoretical framework for understanding LQLS performance under various signal and noise models.
• Findings highlight the nuanced role of predictor correlations in sparse signal recovery.
• Analytical predictions are validated by extensive computational experiments.