Density-preserving sampling: robust and efficient alternative to cross-validation for error estimation.

Area of Science:

• Machine Learning
• Statistical Modeling

Background:

• Estimating generalization ability is crucial for model selection and predicting performance on new data.
• Current methods like cross-validation (CV) are stochastic, requiring multiple runs and significant computation.
• This computational expense can be prohibitive for complex models or large datasets.

Purpose of the Study:

• To propose a new, computationally efficient method for generalization error estimation.
• To eliminate the need for repeated runs in error estimation procedures.
• To provide reliable and accurate error estimates comparable to repeated CV.

Main Methods:

• Developed a correntropy-inspired density-preserving sampling (DPS) procedure.
• DPS divides data into representative subsets, ensuring accuracy without repetition.
• The method was validated using benchmark datasets and standard classifiers.

Main Results:

• DPS produces low-variance generalization error estimates.
• Accuracy is comparable to 10-fold repeated CV.
• Achieves these results at a fraction of the computational cost of CV.

Conclusions:

• The DPS procedure offers a computationally efficient and accurate alternative for generalization error estimation.
• DPS is suitable for model ranking and selection.
• This method addresses the limitations of traditional stochastic estimation techniques.