Empirical simulation of internal validation methods for prediction models: comparing k-fold cross-validation with...

Area of Science:

• Biostatistics
• Machine Learning
• Clinical Informatics

Background:

• Internal validation of clinical prediction models is crucial but methods remain debated.
• K-fold cross-validation and bootstrap methods are commonly recommended but their performance across diverse models is unclear.

Purpose of the Study:

• To systematically evaluate k-fold cross-validation and bootstrap-based optimism correction for internal validation.
• To compare their performance across various statistical and machine learning models using simulated clinical data.

Main Methods:

• Simulated 239,415 inpatient data, developing prediction models for acute kidney injury (AKI) using logistic regression, SVM, Random Forest, etc.
• Applied 2-fold, 5-fold, and 10-fold cross-validation and bootstrap estimators (.632, .632+) for internal validation.
• Assessed accuracy by comparing 1000 AUC estimates against a reference dataset AUC.

Main Results:

• For parametric models, the .632+ estimator showed high accuracy, with 10-fold cross-validation exhibiting minimal bias.
• Bootstrap methods significantly overestimated AUC for nonparametric models, irrespective of sample size.
• 10-fold cross-validation demonstrated consistent, good performance across all model types and sample sizes.

Conclusions:

• Bootstrap methods' performance varies with model complexity; .632+ is best for parametric models.
• 10-fold cross-validation is more robust, easier to implement, and performs well universally.
• Prioritize 10-fold cross-validation for internal validation of prediction models.