Extrapolated cross-validation for randomized ensembles.

Area of Science:

• Machine Learning
• Computational Biology
• Statistical Modeling

Background:

• Ensemble methods, including bagging and random forests, are widely used across diverse scientific domains.
• Efficient tuning of ensemble parameters remains a significant challenge despite their prevalence.
• Existing cross-validation methods can be computationally intensive or suboptimal for parameter tuning.

Purpose of the Study:

• Introduce Extrapolated Cross-Validation (ECV) for optimizing ensemble and subsample sizes in randomized ensembles.
• Develop a method that achieves high accuracy and computational efficiency in parameter tuning.
• Address the need for effective tuning strategies in high-dimensional data and computational constraints.

Main Methods:

• Utilize out-of-bag errors for initial estimators at small ensemble sizes.
• Employ a novel risk extrapolation technique based on prediction risk decomposition.
• Establish uniform consistency of the risk extrapolation for ensemble and subsample sizes.

Main Results:

• ECV yields $$-optimal ensembles for squared prediction risk, approaching oracle-tuned performance.
• The method demonstrates theoretical consistency across various ensemble and subsample sizes, including high-dimensional settings.
• In a case study predicting surface protein abundances, ECV outperformed sample-split and k-fold cross-validation.

Conclusions:

• ECV offers a computationally efficient and accurate approach for tuning randomized ensembles.
• The method is theoretically robust, accommodating general predictors and mild moment assumptions.
• ECV provides a practical solution for parameter optimization in complex biological data analysis under computational constraints.