Analysis of the ISIC image datasets: Usage, benchmarks and recommendations.

Area of Science:

• Dermatology and Medical Image Analysis
• Machine Learning in Healthcare
• Computational Pathology

Background:

• The International Skin Imaging Collaboration (ISIC) datasets are crucial for machine learning in skin cancer detection.
• These datasets contain numerous dermoscopic images with diagnostic metadata, driving advancements in malignancy assessment.
• Melanoma, though less common, is a serious skin cancer requiring early detection.

Purpose of the Study:

• To analyze the usage and identify irregularities within ISIC datasets from 2016-2020.
• To propose and validate a strategy for removing duplicate images across ISIC datasets.
• To establish benchmark results for melanoma classification using a curated ISIC dataset.

Main Methods:

• Analysis of yearly ISIC dataset releases (2016-2020) to identify duplicate images.
• Development and application of a duplicate removal strategy for training and testing sets.
• Conducting melanoma classification experiments on ISIC 2020 and ISIC 2017 test sets post-data curation.

Main Results:

• A significant number of duplicate images were identified within and between ISIC datasets, including across training and testing splits.
• Removal of 14,310 duplicate images from the training set was performed.
• The best performing model achieved an Area Under the Curve (AUC) of 0.80 for melanoma prediction on the ISIC 2020 test set.

Conclusions:

• Duplicate images present a challenge in ISIC datasets, potentially affecting research reproducibility and model performance.
• A curated dataset with duplicates removed is recommended for more reliable machine learning research in skin cancer.
• The proposed duplicate removal strategy and curated dataset offer a valuable resource for future studies in dermatological image analysis.