Randomized boosting with multivariable base-learners for high-dimensional variable selection and prediction.

Area of Science:

• Computational statistics
• Biomedical data analysis
• Machine learning

Background:

• Statistical boosting traditionally uses single-variable base-learners, which can lead to overfitting and inclusion of noise variables in high-dimensional biomedical data.
• The number of iterations in boosting is often tuned for predictive performance, potentially resulting in overly complex models.

Purpose of the Study:

• To extend classical component-wise gradient boosting with multivariable base-learners for improved model selection and estimation.
• To develop algorithms that automatically stop and yield sparser, more interpretable prediction models for high-dimensional biomedical data.

Main Methods:

• Propose three extensions: Subspace Boosting (SubBoost) with multivariable base-learners and information criteria for stopping.
• Introduce Random Subspace Boosting (RSubBoost) with random preselection for scalability.
• Develop Adaptive Subspace Boosting (AdaSubBoost) with adaptive preselection focusing on predictive base-learners.

Main Results:

• Subspace algorithms are beneficial for data with high correlations among covariates.
• Proposed algorithms yield sparser models compared to classical boosting in biomedical applications.
• Achieve competitive predictive performance against penalized regression methods like lasso and elastic net.

Conclusions:

• Randomized boosting approaches with multivariable base-learners are effective extensions for highly-correlated, sparse, high-dimensional data.
• Information criteria-based selection of base-learners ensures automatic stopping, promoting parsimonious and interpretable models.