Cross-validation pitfalls when selecting and assessing regression and classification models.

Area of Science:

• Computational Chemistry
• Machine Learning
• Statistical Modeling

Background:

• Current cross-validation methods for model selection and assessment can result in high variance, limiting their use in applications like QSAR.
• Improved reliability and confidence in model selection are crucial for practical applications.

Purpose of the Study:

• To describe and evaluate best practices for reliable model selection and assessment using cross-validation.
• To introduce cloud computing as a key component for enabling advanced cross-validation approaches.

Main Methods:

• Detailed algorithm for repeated grid-search V-fold cross-validation for parameter tuning.
• Definition of a repeated nested cross-validation algorithm for model assessment.
• Comparison of repeated grid-search cross-validation and double cross-validation for variable selection and parameter tuning.

Main Results:

• Demonstrated the impact of dataset splits in V-fold cross-validation on prediction performance variation.
• Results on seven QSAR datasets highlight the need to consider performance variation during model selection and assessment.

Conclusions:

• Repeating cross-validation is essential for selecting optimal classification and regression models.
• Repeating nested cross-validation is critical for accurately assessing prediction error.