FusionM4Net: A multi-stage multi-modal learning algorithm for multi-label skin lesion classification.

Area of Science:

• Dermatology and Medical Imaging
• Artificial Intelligence in Healthcare
• Machine Learning for Disease Classification

Background:

• Skin diseases are globally prevalent, necessitating accurate diagnostic tools.
• Deep learning excels at skin lesion recognition, primarily using dermoscopy images.
• Existing multi-modal approaches often use single-stage feature fusion, limiting data integration.

Purpose of the Study:

• To develop a novel two-stage multi-modal learning algorithm (FusionM4Net) for multi-label skin disease classification.
• To enhance diagnostic accuracy by integrating clinical images, dermoscopy images, and patient metadata.
• To improve upon existing methods by employing both feature-level and decision-level data fusion.

Main Methods:

• Proposed FusionM4Net algorithm with a two-stage approach for multi-modal data fusion.
• Stage 1: FusionNet integrates clinical and dermoscopy images at feature and decision levels.
• Stage 2: Fusion Scheme 2 incorporates patient metadata with Stage 1 predictions using an SVM classifier.

Main Results:

• FusionM4Net-FS (Stage 1) achieved 75.7% average accuracy and 74.9% diagnostic accuracy without metadata.
• FusionM4Net-SS (full model) improved performance to 77.0% average accuracy and 78.5% diagnostic accuracy.
• Demonstrated robust performance on a label-imbalanced dataset, outperforming state-of-the-art methods.

Conclusions:

• The proposed FusionM4Net effectively integrates multi-modal data for improved skin disease classification.
• Two-stage fusion, incorporating decision-level fusion and patient metadata, significantly enhances diagnostic accuracy.
• FusionM4Net offers a robust solution for multi-label skin disease classification, particularly on imbalanced datasets.