Distributional bias compromises leave-one-out cross-validation.

Area of Science:

• Machine Learning
• Statistical Modeling
• Data Science

Background:

• Cross-validation is a standard technique for assessing machine learning model generalization.
• Leave-one-out cross-validation (LOOCV) is frequently employed in low-data scenarios.
• Aggregating predictions across LOOCV folds is common practice for performance metrics.

Purpose of the Study:

• To identify and mathematically prove the existence of "distributional bias" in aggregated cross-validation.
• To demonstrate the negative impact of distributional bias on model evaluation and hyperparameter tuning.
• To develop and validate a novel cross-validation approach robust to distributional bias.

Main Methods:

• Theoretical proof establishing the negative correlation between training fold means and test instance labels.
• Empirical validation across diverse machine learning tasks, models, and evaluation metrics.
• Development and simulation of a rebalanced cross-validation technique for bias mitigation.

Main Results:

• Distributional bias was proven to be an inherent artifact of aggregated LOOCV, negatively affecting performance.
• This bias was observed across various machine learning applications and can unfairly penalize strong regularization.
• The proposed rebalanced cross-validation method demonstrated improved accuracy and robustness in simulations and benchmarks.

Conclusions:

• Aggregated leave-one-out cross-validation introduces a systematic distributional bias, compromising evaluation reliability.
• A new rebalanced cross-validation strategy effectively mitigates this bias in both classification and regression.
• This method offers a more accurate and reliable approach to machine learning model assessment, particularly in data-scarce settings.