Measuring the optimistic bias of cross-validation in radiomics

In radiomics, independent external data are often unavailable and cross-validation (CV) schemes are widely used to obtain performance estimates. Simple k-fold CV is preferred over nested CV due to its lower computational cost, although nested CV is known to provide more accurate estimates. In a recent study, it was concluded that for practical reasons, there is no significant difference between the two schemes. However, that study was conducted only on low-dimensional datasets, which do not reflect typical radiomic data characteristics. While previous studies suggest that an optimistic bias, in which the true performance of the model is overestimated, might occur in high-dimensional datasets when only simple k-fold CV is used, the extent of this bias in radiomic datasets is currently unknown. This study evaluated whether nested CV is necessary in radiomics by comparing simple k-fold CV, holdout CV, and nested CV across 32 public radiomic datasets. In total, 30,720 models were evaluated to ensure a robust and comprehensive comparison. Each validation scheme was tested by repeatedly splitting the data using a 50:50 train-test ratio. The training data was used for model development and internal validation, whereas the test data was reserved for performance evaluation. Model performance was measured using the area under the receiver operating characteristic curve (AUC), F1-score, and Matthews correlation coefficient (MCC). Cross-validation estimates were then compared to the estimates on the test set to assess potential optimistic bias. In addition, the experiments were repeated on 110 low-dimensional datasets from the UCI repository, involving 105,600 models to compare findings between high- and low-dimensional data. The results revealed that in radiomic data, simple k-fold CV exhibited significant optimistic bias, with a bias of up to 0.128 in AUC, 0.130 in F1-score, and 0.264 in MCC, whereas holdout CV remained relatively unbiased. Nested CV showed low, but variable bias depending on the model, with a simple ensemble achieving the best performance. Larger sample sizes reduced both optimistic bias and variability, particularly in simple k-fold CV, but no significant relationship was observed with dataset dimensionality or feature count. In contrast, simple k-fold CV was unbiased on the UCI datasets (bias < 0.014 in AUC, < 0.016 in F1, and < 0.014 in MCC), and no clear differences were found between the three CV schemes. Therefore, in radiomics, there is a substantial practical difference between simple k-fold and nested CV. This finding further underscores that results from low-dimensional datasets do not necessarily generalize to radiomics.