On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling...

Area of Science:

• Machine Learning
• Statistical Modeling
• Data Science

Background:

• Model validation is critical for supervised learning generalization.
• Effective data splitting strategies are essential for reliable model validation.
• Previous studies have explored various data splitting techniques with mixed results.

Purpose of the Study:

• To comparatively evaluate diverse data splitting methods for supervised model validation.
• To assess the impact of dataset size and training/validation set balance on performance estimation.
• To identify optimal data splitting strategies for robust model generalization.

Main Methods:

• Generated nine simulated datasets using the MixSim model with varying mis-classification probabilities and sample sizes.
• Applied partial least squares for discriminant analysis (PLS-DA) and support vector machines (SVM) for classification.
• Tested data splitting methods: cross-validation variants, bootstrapping, bootstrapped Latin partition, Kennard-Stone (K-S), and sample set partitioning based on joint X-Y distances (SPXY).

Main Results:

• Dataset size is the primary determinant of generalization performance estimation accuracy.
• Small datasets exhibited significant performance gaps between validation and blind test sets across all methods.
• Balanced training/validation set sizes improved performance estimation reliability, approaching central limit theorem approximations.
• Systematic sampling methods (K-S, SPXY) showed poor performance estimation due to unrepresentative sample sets.

Conclusions:

• Dataset size and balanced splitting are crucial for accurate supervised model validation.
• Careful consideration of data splitting strategies is necessary to avoid misleading performance estimates, especially with limited data.
• Systematic sampling methods may not be suitable for reliable model performance estimation in supervised learning.