Effect of Training Data Volume on Performance of Convolutional Neural Network Pneumothorax Classifiers

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Imaging

Background:

• Deep neural networks (DNNs) require large, labeled datasets for optimal performance.
• Acquiring such datasets in radiology is a significant challenge.
• This study addresses the impact of training dataset size on DNN performance in medical imaging.

Purpose of the Study:

• To investigate the relationship between training dataset size and the performance of deep learning classifiers.
• To establish learning curves for deep learning models in radiology tasks.
• To identify potential diminishing returns in performance with increasing data volume.

Main Methods:

• Merged two open-source datasets (ChestX-ray14, CheXpert) totaling 291,454 chest radiograph images.
• Trained convolutional neural networks (ResNet-50, DenseNet-121, EfficientNet) with stepwise increases in training data.
• Evaluated model performance on an external test set of 525 emergency department chest radiographs.
• Performed learning curve analysis to model the AUCs.

Main Results:

• Model performance (AUC and accuracy) increased rapidly with dataset sizes from 2,000 to 20,000 images.
• Performance continued to improve gradually up to the maximum dataset size of 291,000 images.
• Models trained on the full dataset (291k) showed significantly higher AUCs than those trained on 20k images across all architectures (e.g., ResNet-50: 0.86 vs. 0.95).

Conclusions:

• Established learning curves demonstrate a clear correlation between training data volume and deep learning model performance in radiology.
• The findings suggest a point of diminishing returns for increasing training data, impacting cost-effectiveness.
• Algorithm developers should consider these learning curves to balance performance gains with the high costs of radiology data acquisition and labeling.