A multimodal vision foundation model for clinical dermatology.

Area of Science:

• Artificial Intelligence in Medicine
• Dermatology
• Medical Imaging Analysis

Background:

• Diagnosing skin diseases demands advanced visual skills and multimodal data synthesis.
• Current deep learning models are limited in addressing the complex, multimodal nature of clinical dermatology.
• A need exists for AI models that can integrate diverse imaging modalities for comprehensive dermatological assessment.

Purpose of the Study:

• To introduce PanDerm, a multimodal foundation model for dermatology.
• To evaluate PanDerm's performance across a wide range of dermatological tasks.
• To assess the clinical utility and impact of PanDerm on healthcare providers' diagnostic capabilities.

Main Methods:

• PanDerm was pretrained using self-supervised learning on over 2 million real-world skin disease images.
• The model was trained on data from 11 clinical institutions across 4 imaging modalities.
• Performance was evaluated on 28 diverse benchmarks, including screening, diagnosis, segmentation, and prognosis.

Main Results:

• PanDerm achieved state-of-the-art performance across all 28 evaluated tasks.
• The model often outperformed existing methods, even with limited labeled data (10%).
• Reader studies demonstrated PanDerm's superiority in early melanoma detection, diagnostic accuracy, and differential diagnosis support.

Conclusions:

• PanDerm demonstrates significant potential to improve patient care in diverse clinical dermatology scenarios.
• The model's multimodal approach sets a precedent for developing foundation models in other medical specialties.
• PanDerm shows promise in accelerating the integration of AI-driven diagnostic support in healthcare.