nestedcv: an R package for fast implementation of nested cross-validation with embedded feature selection designed...

Area of Science:

• Biomedical data analysis
• Machine learning in healthcare
• Statistical modeling

Background:

• Machine learning models are prevalent in medical research.
• Standard practice often involves simple training/testing splits with cross-validation (CV) for hyperparameter tuning.
• Biomedical data frequently presents challenges with limited sample sizes and a high number of predictors (P >> n).

Purpose of the Study:

• Introduce the nestedcv R package for robust machine learning analysis in biomedical research.
• Address the limitations of standard CV in high-dimensional, low-sample-size scenarios.
• Provide a tool for unbiased performance estimation and feature selection.

Main Methods:

• Implementation of fully nested k x l-fold CV for regularized linear models (glmnet) and other models (caret framework).
• Integration of inner CV for model tuning and outer CV for unbiased performance assessment.
• Inclusion of fast, nested filter functions for feature selection to prevent information leakage.
• Application of outer CV for Bayesian linear and logistic regression with horseshoe priors for model sparsity and unbiased accuracy measurement.

Main Results:

• The nestedcv R package facilitates nested k x l-fold CV for various machine learning models.
• It enables unbiased performance evaluation by nesting feature selection and hyperparameter tuning within the outer CV loop.
• The package supports Bayesian regression models, promoting sparse models and accurate performance metrics.

Conclusions:

• The nestedcv R package offers a comprehensive solution for applying nested cross-validation in machine learning for biomedical data.
• It effectively handles feature selection and model tuning, ensuring unbiased performance estimation.
• This tool is valuable for researchers working with high-dimensional, low-sample-size datasets in medical research.