Classification performance bias between training and test sets in a limited mammography dataset

Area of Science:

• Medical imaging analysis
• Radiomics and machine learning

Background:

• Radiomics studies in mammography are crucial for cancer diagnosis.
• Limited dataset sizes in medical imaging can introduce bias.
• Performance bias from data sampling affects model reliability.

Purpose of the Study:

• To assess performance bias in mammography radiomics due to data splitting.
• To evaluate the impact of training and test set selection on model outcomes.
• To investigate strategies for mitigating bias in limited medical imaging datasets.

Main Methods:

• Utilized mammograms from 700 women for ductal carcinoma in situ upstaging.
• Repeatedly split data into training (n=400) and test (n=300) sets 40 times.
• Employed logistic regression and support vector machines with radiomics and clinical features.

Main Results:

• Observed significant variability in Area Under the Curve (AUC) across data splits.
• Identified a performance tradeoff: improved training accuracy sometimes decreased test accuracy.
• Cross-validation reduced variability but required over 500 cases for representative estimates.

Conclusions:

• Data sampling bias can lead to inappropriate conclusions in mammography radiomics.
• Models trained on non-representative subsets may not generalize well.
• Developing optimal test set selection is critical for reliable medical imaging research.