Assessing Open-world Foundation Models for Zero-shot Skin Segmentation in Clinical Dermatological Photographs

Skin segmentation from clinical photography is a crucial step in dermatological image analysis. However, the variability in skin tones, lighting conditions, anatomical regions, and the presence of additional objects introduces significant challenges. Due to these complexities, the segmentation process is often performed manually, as developing an algorithm capable of handling such diverse conditions is particularly difficult. Recently, open-world foundation models have emerged, offering the potential to generalize across diverse and unseen conditions. These models present a promising opportunity for dermatology. In this work, we adopt two such models-Grounding DINO and SAM 2-to construct a pipeline for zero-shot skin segmentation in dermatology. We evaluated our approach on two clinical skin photography datasets comprising 27,378 images. Based on a manual rating protocol, 77.1% of the segmentations were deemed acceptable, demonstrating robustness in handling real-world clinical photographs. Our results highlight the potential of open-world foundation models to address a challenging problem in dermatology with minimal human involvement.