Analysis: Flawed Datasets of Monkeypox Skin Images.

Area of Science:

• Dermatology
• Computer Science
• Medical Imaging

Background:

• A widely used dataset of Monkeypox skin images was created via web-scrapping from non-medical sources.
• Subsequent research utilized this dataset to develop machine learning (ML) models for diagnosing viral skin lesions.
• Several studies published in peer-reviewed journals claimed high performance for these ML models.

Purpose of the Study:

• To analyze the foundational study that popularized the Monkeypox image dataset.
• To conduct rebuttal experiments to assess the validity and reliability of ML models trained on this dataset.
• To highlight the potential risks associated with using non-medically curated datasets in medical AI development.

Main Methods:

• Critical analysis of the original web-scrapping methodology used to create the Monkeypox image dataset.
• Development and testing of ML models using the problematic dataset.
• Comparative experiments to demonstrate that model performance may not correlate with medically relevant features.

Main Results:

• The dataset contains medically irrelevant images, compromising its utility for diagnosing Monkeypox and similar viral infections.
• ML models trained on this dataset exhibit performance that is not necessarily linked to accurate disease identification.
• The foundational study has spurred the development of numerous unreliable ML diagnostic tools.

Conclusions:

• The reliance on non-curated, medically irrelevant image datasets poses significant risks to the development of trustworthy AI in healthcare.
• Methodological rigor and data provenance are crucial for building reliable computer-aided diagnosis systems for skin conditions.
• Further research should prioritize the use of validated, medically relevant datasets for training diagnostic ML models.