Assessing Open-world Foundation Models for Zero-shot Skin Segmentation in Clinical Dermatological Photographs

Area of Science:

• Dermatology
• Artificial Intelligence
• Medical Imaging

Background:

• Skin segmentation in clinical photography is vital for dermatological image analysis.
• Variability in skin tones, lighting, and anatomy presents significant segmentation challenges.
• Manual segmentation is common due to the difficulty of algorithmic solutions.

Purpose of the Study:

• To develop an automated zero-shot skin segmentation pipeline for dermatology.
• To leverage open-world foundation models for improved generalization in dermatological image analysis.
• To assess the performance of a novel AI pipeline on diverse clinical skin photographs.

Main Methods:

• Utilized Grounding DINO and Segment Anything Model 2 (SAM 2) for zero-shot skin segmentation.
• Constructed an AI pipeline integrating these open-world foundation models.
• Evaluated the pipeline on two large-scale clinical skin photography datasets (27,378 images).

Main Results:

• Achieved 77.1% acceptable segmentations based on a manual rating protocol.
• Demonstrated robustness in handling real-world clinical photographs with diverse conditions.
• Highlighted the effectiveness of foundation models in addressing complex dermatological segmentation tasks.

Conclusions:

• Open-world foundation models show significant potential for automated skin segmentation in dermatology.
• The developed pipeline offers a promising solution with minimal human involvement.
• This approach can advance dermatological image analysis by overcoming traditional segmentation challenges.