Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation

Area of Science:

• Machine Learning
• Statistical Modeling
• Computational Statistics

Background:

• Cross-validation (CV) is crucial for selecting optimal model configurations and estimating predictive performance.
• The performance of the best-selected configuration via CV is often optimistically biased.
• Existing bias correction methods like nested CV can be computationally intensive.

Purpose of the Study:

• To introduce an efficient bootstrap method, Bootstrap Bias Corrected CV (BBC-CV), to correct for optimistic bias in cross-validated performance estimates.
• To develop an accelerated CV method, Bootstrap Bias Corrected with Dropping CV (BBCD-CV), by incorporating a bootstrap-based criterion for early termination of inferior models.
• To compare the proposed methods with existing techniques in terms of computational efficiency, bias, and variance.

Main Methods:

• Bootstrapping the entire model selection process using out-of-sample predictions to correct for bias without retraining.
• Implementing a bootstrap-based statistical criterion to identify and cease training for likely inferior model configurations, leading to computational savings.
• Applying the methods to various performance metrics, including accuracy, AUC, concordance index, and mean squared error.

Main Results:

• BBC-CV effectively corrects the optimistic bias in cross-validated performance estimates.
• BBC-CV demonstrates greater computational efficiency, reduced bias, and smaller variance compared to nested CV and other alternatives.
• BBCD-CV further enhances efficiency by enabling early stopping of suboptimal model training while maintaining accurate performance estimation.

Conclusions:

• BBC-CV offers a computationally efficient and statistically sound approach to obtain unbiased performance estimates for machine learning model selection.
• BBCD-CV provides a practical and accelerated alternative for cross-validation, especially in large-scale or computationally demanding scenarios.
• The proposed bootstrap-based methods are versatile and applicable across a wide range of predictive modeling tasks and performance metrics.