Multimodal skin lesion classification using deep learning.

Area of Science:

• Dermatology
• Artificial Intelligence
• Medical Imaging

Background:

• Convolutional Neural Networks (CNNs) show promise in skin lesion classification.
• Previous CNN studies often rely on single macroscopic images and binary outputs.
• Integrating diverse data sources can enhance diagnostic performance.

Purpose of the Study:

• To develop and evaluate a multimodal classification method for automated skin lesion diagnosis.
• To compare the performance of the multimodal approach against single-modality classifiers.
• To assess the diagnostic utility of dermatoscopic versus macroscopic images.

Main Methods:

• A novel method combining dermatoscopic images, macroscopic images, and patient metadata was developed.
• The multimodal classifier was evaluated on both binary (melanoma detection) and five-class classification tasks.
• Performance was benchmarked against a baseline classifier using only macroscopic images.

Main Results:

• The multimodal classifier achieved higher performance in binary melanoma detection (AUC 0.866 vs 0.784) and multiclass classification (mAP 0.729 vs 0.598).
• Dermatoscopic images yielded superior automated diagnosis performance compared to macroscopic images.
• Experiments were conducted on a dataset comprising 2917 cases with multiple data types.

Conclusions:

• Combining multiple imaging modalities and patient metadata significantly enhances automated skin lesion diagnosis.
• Dermatoscopic imaging offers a performance advantage over macroscopic imaging for automated skin lesion classification.
• The proposed multimodal approach represents a promising advancement for clinical decision support in dermatology.