Assessing the Generalizability of Deep Learning Models Trained on Standardized and Nonstandardized Images and Their...

Area of Science:

• Artificial Intelligence
• Dermatology
• Medical Imaging

Background:

• Convolutional neural networks (CNNs) show potential for skin cancer diagnosis.
• Current CNN models often use non-standardized retrospective image data, limiting their real-world application.

Purpose of the Study:

• To evaluate the impact of standardized versus non-standardized training data on CNN performance for skin cancer classification.
• To assess the generalizability of CNN models across different populations and image datasets.

Main Methods:

• Trained three CNNs of identical architecture: one on non-standardized images (CNN-NS) and two on standardized images (CNN-S, CNN-S2).
• Tested models on three external datasets (Danish, ISIC 2020, UQ) to measure sensitivity, specificity, and AUROC.
• Compared CNN performance against teledermatologist assessments on the Danish dataset.

Main Results:

• Standardized CNN models (CNN-S, CNN-S2) significantly outperformed the non-standardized model (CNN-NS) across all test datasets (P<.001 to P=.009).
• Teledermatologists surpassed CNN model performance, but differences between teledermatologists and the best standardized CNN (CNN-S) were not statistically significant.
• Image quality impacted the performance of all CNN models and teledermatologists.

Conclusions:

• CNNs trained with standardized image data demonstrate superior performance and generalizability for skin cancer classification.
• Standardization of training data is crucial for developing robust and reliable AI diagnostic tools in dermatology.
• Findings have implications for future AI algorithm development, regulation, and clinical approval.