Classification performance bias between training and test sets in a limited mammography dataset.

Area of Science:

• Medical Imaging Analysis
• Radiomics
• Machine Learning in Healthcare

Background:

• Clinical datasets in medical imaging are often limited in size.
• Models trained on different subsets may not represent the entire dataset.
• Performance bias from data sampling can lead to erroneous conclusions.

Approach:

• Assessed performance bias from data sampling in a mammography radiomics study.
• Used 700 mammograms, repeatedly splitting into training (n=400) and test (n=300) sets 40 times.
• Employed logistic regression and support vector machines with radiomics and clinical features.

Key Points:

• Area under the curve (AUC) performance varied significantly across data splits.
• Regression models showed a performance tradeoff between training and testing.
• Cross-validation reduced variability but required large sample sizes (500+) for representative estimates.

Conclusions:

• Data sampling strategies significantly impact model performance and reliability.
• Potential for performance bias necessitates development of optimal test set selection methods.
• Ensuring representative data splits is vital for the clinical significance of radiomics findings.