Related Experiment Video
Updated: May 13, 2026
Guidelines and Experience Using Imaging Biomarker Explorer (IBEX) for Radiomics
Published on: January 8, 2018
Decomposing performance inflation in radiomics: factorial quantification of analytical leakage
Clemente García-Hidalgo1, Jose Antonio Consentino Hernández1, Jose Vicente Cayuela Espí1
1Servicio de Radiología, Hospital General Universitario Morales Meseguer, Avenida Marqués de los Vélez s/n, 30008 Murcia, Spain.
Purpose:
To quantify the cumulative performance inflation produced by analytical decisions leaked outside nested cross-validation in radiomics, and to decompose it by decision type.
Methods:
We compared six pipeline configurations with progressively leaked analytical decisions using Monte Carlo simulation (3 sample sizes x 3 feature dimensions x 2 scenarios, 100 repetitions each) and validated findings on real radiomic features from The Cancer Imaging Archive (TCIA) NSCLC-Radiomics dataset (n = 726, 111 features, 30 permutation-null repetitions). The primary outcome was AUC inflation (fully leaked pipeline minus properly nested pipeline) with bootstrap 95% confidence intervals. Secondary outcomes included false positive rate, additive decomposition by decision type, and concordance between synthetic and real-data profiles (Spearman rho, ICC).
Results:
Inflation was statistically significant across all 19 tested conditions (all p < 0.001). At n = 50 and p = 1,000, the fully leaked pipeline returned a mean AUC of 1.000 [95% CI: 1.000-1.000] on pure noise, versus 0.493 [0.472-0.514] for the nested reference, a difference of 0.507 [0.486-0.528]. Feature selection leakage accounted for the bulk of this effect (60-92% across null and weak-signal conditions), outweighing hyperparameter tuning by a factor of 4 to 8. The same staircase pattern appeared on real radiomic features (Spearman rho = 0.943, p = 0.005). Every leaked-pipeline experiment yielded an AUC above 0.60, in every synthetic condition, even though no signal existed.
Conclusion:
Leaked analytical decisions inflate radiomics performance cumulatively and severely, with feature selection leakage alone driving most of the effect. On real data, leaked feature selection paradoxically lowered generalization performance. All analytical decisions must be enclosed within the validation loop. Studies should report their full analytical search space, and reviewers should verify that feature selection, hyperparameter tuning, and model comparison were all nested.