Enhancing semi-supervised skin lesion segmentation with text-guided pseudo descriptions

Accurate segmentation of skin lesions in dermoscopy images remains challenging due to the scarcity of densely annotated medical images, as manual pixel-level annotation demands significant expert effort. Conversely, textual descriptions are more readily obtainable and can potentially provide rich semantic context for improving segmentation models. In this paper, we propose a novel framework leveraging text-guided pseudo descriptions to enhance semi-supervised skin lesion segmentation. Our approach capitalizes on the emerging capabilities of vision large-language models to generate approximate textual descriptions of unlabeled dermoscopy images, which, despite being noisy, contain valuable semantic information. We also introduce a novel image-text feature fusion mechanism that effectively integrates visual features with textual semantic. To handle the inherent high noise levels in dermoscopy images, we develop a specialized data augmentation strategy specifically designed for skin lesion characteristics. Experimental results on public benchmark datasets demonstrate that our text-guided approach significantly improves segmentation performance compared to traditional semi-supervised methods, particularly in low-annotation settings. Moreover, the textual descriptions provide interpretable insights into the segmentation process, offering clinically relevant context beyond pixel-level predictions. From a long-term perspective, our work suggests that combining minimal image annotations with more easily obtained textual descriptions presents a more efficient and effective approach to medical image segmentation. To facilitate future research, we release the ISIC-GPT dataset, a novel text-image paired dataset based on ISIC 2017 and 2018 challenges.