A benchmark for neural network robustness in skin cancer classification

Area of Science:

• Dermatology
• Medical Imaging
• Artificial Intelligence

Background:

• Deep learning classifiers are widely used for skin cancer classification on dermoscopic images.
• Current evaluation methods using holdout data may not reveal limitations like susceptibility to confounding factors.
• Thorough evaluation on out-of-distribution (OOD) data is crucial for clinical applicability.

Purpose of the Study:

• To establish a dermoscopic skin cancer benchmark for measuring classifier robustness to OOD data.
• To provide a standardized method for evaluating the performance of AI models in real-world scenarios.

Main Methods:

• Created an OOD robustness benchmark using a proprietary dermoscopic image database and image transformations.
• Evaluated the robustness of four different convolutional neural network (CNN) architectures.
• Developed three datasets: Skin Archive Munich (SAM), SAM-corrupted (SAM-C), and SAM-perturbed (SAM-P).

Main Results:

• All four evaluated CNNs demonstrated susceptibility to image corruptions and perturbations.
• SAM-C tested classifier performance against low-quality inputs.
• SAM-P measured prediction stability across minor image variations.

Conclusions:

• The developed benchmark facilitates OOD testing for binary skin cancer classifiers.
• Classifier performance confirmed existing shortcomings of CNNs, providing a reference point.
• This benchmark aims to improve the development of more robust skin cancer classification models.