Externally validated yet undertrained: sample size deficits in machine learning-based radiomics.

Area of Science:

• Radiomics and Machine Learning
• Medical Imaging Analysis
• Biostatistics

Background:

• Machine learning (ML) models in radiomics are increasingly used for medical image analysis.
• External validation is crucial for assessing the generalizability of these models.
• High-impact journals often publish studies on externally validated radiomics models.

Purpose of the Study:

• To systematically evaluate the adequacy of training sample sizes in externally validated ML-based radiomics models.
• To quantify the discrepancy between current practices and theoretical minimum sample size requirements.
• To assess the impact of sample size on model stability and reproducibility.

Main Methods:

• A systematic review of ML-based radiomics models published in Q1 journals between January 2023 and August 2025.
• Inclusion criteria: binary prediction models, ML algorithms (excluding logistic regression), external validation.
• Application of a sample size framework to calculate minimum required training instances based on model complexity, outcome prevalence, and feature count.

Main Results:

• 16% of studies were excluded due to missing data essential for sample size estimation.
• In the final cohort (n=28), 90% of models had inadequate training sample sizes, with a median deficit of 195.5 instances.
• Only 10.7% of studies met criteria for stable model development; most failed basic heuristics like 10 events per predictor.

Conclusions:

• The majority of externally validated radiomics models in high-impact journals are trained on statistically insufficient datasets.
• This data deficit leads to overfitting and instability, potentially explaining the radiomics reproducibility crisis.
• Insufficient sample sizes undermine model reliability and can lead to misinformed clinical decisions.