Distributional bias compromises leave-one-out cross-validation

Cross-validation is a common method for evaluating machine learning models. "Leave-one-out cross-validation," in which each data instance is used to test a model trained on all other instances, is often used in data-scarce regimes. As common metrics such as the R² score cannot be calculated for a single prediction, predictions are commonly aggregated across folds for performance evaluation. Here, we prove that this creates "distributional bias": a negative correlation between the average label of each training fold and the label of its corresponding test instance. As machine learning models tend to regress to the mean of their training data, this bias tends to negatively affect performance evaluation and hyperparameter optimization. We demonstrate that distributional bias exists across diverse tasks, models, and evaluation approaches, and can bias against stronger regularization. To address it, we developed a generalizable rebalanced cross-validation that is robust to distributional bias in both classification and regression, and demonstrates improved performance in simulations, machine learning benchmarks, and several published analyses.