Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles

Area of Science:

• Machine Learning
• Statistical Learning Theory
• Ensemble Methods

Background:

• Feature bagging is a standard technique for reducing prediction variance in machine learning by training estimators on feature subsets.
• Understanding its theoretical underpinnings, particularly in the context of noisy data and complex model behaviors like double descent, is crucial for optimizing its application.
• Ridge ensembles are commonly used for their regularization properties in linear regression problems.

Approach:

• Develop a theoretical framework for feature bagging within noisy least-squares ridge ensembles.
• Derive and simplify analytical learning curves, especially for equicorrelated data.
• Investigate the impact of feature subsampling on the double-descent phenomenon in linear predictors.
• Introduce and analyze heterogeneous feature ensembling as a method to combat double descent.
• Compare the performance of feature-subsampling ensembles against single linear predictors, analyzing noise trade-offs.
• Extend qualitative insights to linear classifiers on image classification tasks using deep learning feature maps.

Key Points:

• Feature bagging in noisy ridge ensembles is theoretically analyzed.
• Subsampling features demonstrably shifts the double-descent peak of linear predictors.
• Heterogeneous feature ensembling offers a computationally efficient strategy to mitigate double descent.
• A trade-off exists between noise amplification from subsampling and noise reduction from ensembling.
• The findings are validated for linear classifiers on deep learning-derived image features.

Conclusions:

• Feature bagging theory is extended to noisy ridge ensembles, revealing insights into learning curve behavior.
• Heterogeneous feature ensembling is proposed as an effective and efficient method to address the double-descent problem.
• The study provides a nuanced understanding of the performance trade-offs in feature bagging, applicable from linear models to complex image classification tasks.