Variational Garrote for Statistical Physics-based Sparse and Robust Variable Selection

Area of Science:

• Statistical physics
• Machine learning
• Data science

Background:

• High-dimensional data analysis requires effective variable selection.
• Sparse regression methods like Ridge and LASSO are widely used but have limitations.
• The statistical physics-based Variational Garrote (VG) is an underutilized method for feature selection.

Purpose of the Study:

• To enhance the Variational Garrote (VG) method using automatic differentiation for improved scalability and efficiency.
• To evaluate the performance of the enhanced VG method on synthetic and real-world datasets.
• To investigate the behavior of VG in highly sparse regimes and its ability to identify the correct number of relevant variables.

Main Methods:

• Revisiting and enhancing the statistical physics-based Variational Garrote (VG) method.
• Incorporating automatic differentiation techniques for efficient optimization.
• Evaluating performance on synthetic datasets with controlled sparsity and real-world complex datasets.

Main Results:

• The enhanced VG method demonstrates robust and consistent variable selection, outperforming Ridge and LASSO regression, especially in highly sparse data.
• A sharp transition point was identified, indicating abrupt degradation in generalization and increased selection uncertainty as superfluous variables are added.
• This transition point serves as a practical signal for estimating the optimal number of relevant variables.

Conclusions:

• The enhanced Variational Garrote (VG) offers a powerful and scalable approach for sparse modeling and feature selection.
• VG provides a reliable method for identifying key predictors in high-dimensional data.
• The identified transition point offers valuable insights for model selection and understanding data complexity across various applications.