Radiomics machine learning study with a small sample size: Single random training-test set split may lead to...

Area of Science:

• Radiomics
• Machine Learning
• Medical Imaging Analysis

Background:

• Machine learning models in radiomics are sensitive to data splitting strategies.
• Estimating model performance requires robust validation to avoid bias.

Purpose of the Study:

• To investigate the impact of random training-test set splitting on machine learning model performance estimation in brain tumor radiomics.
• To evaluate the generalization gap under different data conditions and task difficulties.

Main Methods:

• Utilized magnetic resonance imaging (MRI) radiomics features for two classification tasks: glioblastoma vs. brain metastasis and low- vs. high-grade meningiomas.
• Performed 1,000 random training-test splits on original and undersampled datasets.
• Trained a least absolute shrinkage and selection operator (LASSO) model and evaluated performance using the area under the curve (AUC).

Main Results:

• AUC varied significantly across different training-test splits, particularly for difficult tasks and undersampled data.
• The mean AUC difference (generalization gap) was higher in difficult tasks (0.092 ± 0.071 with undersampling) compared to simple tasks (0.039 ± 0.032 without undersampling).
• Validation methods did not sufficiently reduce the generalization gap when it was large.

Conclusions:

• Single random training-test splits can lead to unreliable performance estimates in radiomics research.
• The findings highlight the critical need for careful data splitting and validation strategies, especially with limited sample sizes.
• Machine learning applications in radiomics require robust methodologies to ensure reliable and reproducible results.