Characterizing the Impact of Training Data on Generalizability: Application in Deep Learning to Estimate Lung Nodule...

Area of Science:

• Artificial intelligence in medical imaging
• Machine learning for disease detection
• Radiology and oncology research

Background:

• Pulmonary nodules on low-dose CT scans require accurate malignancy risk assessment for lung cancer screening.
• Deep learning algorithms show promise in improving diagnostic accuracy.
• Understanding the impact of training data volume is crucial for AI development.

Purpose of the Study:

• To investigate the relationship between training data volume and the performance of a deep learning AI for pulmonary nodule malignancy risk assessment.
• To determine the minimum training data required for AI performance to be non-inferior to models trained on full datasets and to clinician performance.

Main Methods:

• Retrospective study using 16,077 annotated nodules from the National Lung Screening Trial (NLST).
• Systematic training of a deep learning AI algorithm on stratified NLST data subsets (1.25% to 100%).
• External validation using the Danish Lung Cancer Screening Trial (DLCST) data for performance comparison.

Main Results:

• The AI achieved a mean AUC of 0.92 on the DLCST cohort when trained on the full NLST dataset.
• Non-inferior performance was maintained with 80% of NLST training data (mean AUC 0.92).
• Clinician-level performance (mean AUC 0.82) was achieved with only 20% of the training data on a size-matched DLCST subset.

Conclusions:

• Deep learning AI demonstrates excellent performance in assessing pulmonary nodule malignancy risk.
• Clinical-level performance can be achieved with significantly less training data than initially anticipated.
• Peak AI performance was reached before utilizing the entire available training dataset.