Lifting the radiomics performance ceiling: how preprocessing leakage amplifies model multiplicity

To jointly quantify and causally link preprocessing leakage, analytical flexibility and the Rashomon effect in radiomics-based clinical prediction models.

Retrospective methodological (specification-curve) analysis of 50 de-identified binary-classification datasets in radMLBench (multicentre; 13 anatomical sites; CT, MRI, PET/CT; median 147 patients per dataset, range 51-969); prediction-model reporting followed CLAIM 2.0 and TRIPOD + AI 2024. We enumerated an analytical multiverse of 456 pipeline specifications (3 scalers × 2 correlation filters × 19 feature selectors × 4 classifiers), each evaluated against the dataset label by 5-fold cross-validation × 10 repetitions. Feature-level preprocessing leakage was assessed by paired global-versus-nested comparison for the synthetic minority oversampling technique (SMOTE), normalisation, correlation filtering and principal component analysis (PCA), pooled by DerSimonian-Laird meta-analysis. Within-subjects causal mediation (43 datasets) tested whether inflation of the best area under the receiver operating characteristic curve (AUC) explains leakage-induced Rashomon-set expansion.

Across the 50 datasets, SMOTE leakage inflated AUC by a pooled 0.09 (95% confidence interval (CI): 0.07, 0.11; p < 0.001), exceeding 0.05 in 64%. Pipeline choice shifted AUC by 0.30, classifier dominating (eta-squared = 0.21). At ε = 5%, near-optimal pipelines disagreed on 20-30% of patient classifications (Fleiss kappa = 0.58) and shared < 30% of top features. SMOTE leakage quadrupled the Rashomon set (0.06 to 0.23; p < 0.001); 79% mediated by best-AUC inflation. Radiomics showed 4.8× wider vibration of effects (VoE) but 13× smaller Rashomon ratios than eight tabular benchmarks (both p < 0.001).

The three sources compound; leakage amplifies model multiplicity by inflating the performance ceiling.

Question Whether preprocessing leakage, analytical flexibility and the Rashomon effect jointly compromise the reproducibility of radiomics-based clinical prediction models has not been quantified. Findings Across 50 radiomics datasets, SMOTE leakage inflated AUC by 0.09 and quadrupled the Rashomon set, with 79% causally mediated by best-AUC inflation. Clinical relevance Radiomics-derived clinical prediction models routinely disagree on 20-30% of patients while appearing equally near-optimal. The released three-layer audit tool quantifies this risk per study and should precede any clinical deployment claim.