Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions...

Area of Science:

• Dermatology and Medical Artificial Intelligence
• Computational Pathology and Machine Learning

Background:

• Early detection of skin cancer is critical for patient outcomes but often hindered by a lack of clinician training.
• Machine learning (ML) offers a promising avenue to enhance diagnostic capabilities in primary care settings.

Purpose of the Study:

• To develop a neural network for classifying skin lesions as malignant or benign.
• To utilize a multimodal approach combining smartphone images and clinical data with transfer learning.

Main Methods:

• Developed three neural network models: clinical data-based, image-based (DenseNet-121), and multimodal.
• Employed Bayesian Optimisation HyperBand for model tuning and 5-fold cross-validation.
• Evaluated performance using AUC-ROC, Brier score, MCC, sensitivity, and specificity; explored explainability with permutation importance and Grad-CAM.

Main Results:

• The multimodal network achieved an AUC-ROC of 0.91 during cross-validation and internal validation.
• Multimodal model outperformed unimodal models on key metrics, demonstrating robust classification performance.
• Explainability analyses identified influential clinical features and confirmed image-based model focus on lesion characteristics.

Conclusions:

• A transfer learning-based multimodal neural network effectively identifies malignant skin lesions using smartphone images and clinical data.
• The model shows potential for improving early skin cancer detection in primary care.
• Further external validation on diverse datasets is recommended to ensure generalizability and support clinical integration.