The challenge of data scarcity and imbalanced classes in radiomics performance

Radiomics-based machine learning models are increasingly used for clinical decision-making, yet their reliability is often constrained by limited sample sizes and class imbalance in imaging datasets. While prior studies addressed these challenges independently, their combined impact remains insufficiently explored. This study quantifies the individual and joint effects of progressive data scarcity and class imbalance on radiomics model performance, aiming to identify mitigation strategies.

Three radiomics datasets-PI-CAI (clinically significant prostate cancer detection), BraTS2021 (MGMT promoter methylation prediction in glioblastoma), and Hunter2023 (lung nodule malignancy classification)-were analyzed under four conditions: baseline (balanced, fixed-size dataset), progressive class imbalance, progressive sample size reduction, and a combined scenario. Five machine learning models were evaluated, with XGB selected for PI-CAI and Hunter2023 and MLP for BraTS2021 based on highest balanced accuracy. Imbalance was addressed using state-of-the-art sampling techniques, and data scarcity was mitigated using Tabular Variational Autoencoders (TVAE). Statistical significance was evaluated using paired permutation t-tests across repeated cross-validation splits.

Feature selection played a key role in model performance and interpretability. The most predictive features were biologically plausible and dataset-specific, such as perinodular texture heterogeneity in lung cancer or gray-level non-uniformity in glioblastoma. Under progressive class imbalance, unsampled models showed degradation in balanced accuracy. Applying the best-performing sampling strategy improved balanced accuracy in PI-CAI (Δ = 0.065; dz = 1.54), BraTS2021 (Δ = 0.057; dz = 1.25), and Hunter2023 (Δ = 0.087; dz = 1.35), all p < 0.001. Under progressive sample size reduction, TVAE showed dataset-dependent effects: no significant change in PI-CAI, moderate degradation in BraTS2021 (Δ = -0.042; dz = -0.60; p < 0.001), and improvement in Hunter2023 (Δ = 0.029; dz = 0.46; p < 0.001). In the combined scenario, TVAE plus sampling increased balanced accuracy in PI-CAI (Δ = 0.023; dz = 0.42; p = 0.03), BraTS2021 (Δ = 0.070; dz = 0.95; p < 0.001), and Hunter2023 (Δ = 0.041; dz = 0.61; p < 0.01).

Class imbalance consistently impairs radiomics performance and may be mitigated through undersampling-based strategies. The impact of augmentation under data scarcity is dataset-dependent, but combined correction strategies yield gains under compounded constraints.