Sketch-a-Segmenter: Sketch-based Photo Segmenter Generation.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Traditional photo segmentation models require extensive pixel-level annotated data for specific object categories.
• Existing models are limited to pre-defined categories, hindering their application to novel or unseen objects.
• Few-shot and zero-shot learning have emerged as solutions for image segmentation with limited data.

Purpose of the Study:

• To demonstrate the efficacy of using sketches as a transferable representation for photo segmentation.
• To develop a novel sketch-based method for generating photo segmentation models for new categories.
• To achieve more detailed and accurate segmentation by leveraging the fine-grained nature of sketches.

Main Methods:

• A sketch-based photo segmentation framework is proposed, which synthesizes neural network weights from sketch inputs.
• The framework is designed to operate at both category-level and instance-level segmentation.
• A new dataset, SketchySeg, was created, pairing sketches with corresponding photo segmentation annotations.

Main Results:

• The proposed method successfully generates photo segmentation models for novel categories using only a single sketch.
• Fine-grained sketches lead to more accurate instance-level segmentation.
• The framework demonstrates generalization across categories, offering an alternative to zero-shot learning.

Conclusions:

• Sketches serve as a valuable and transferable representation for photo segmentation, particularly for novel categories.
• Sketch-based photo segmentation provides a flexible and efficient approach, outperforming traditional methods in scenarios with limited annotated data.
• The developed framework and dataset advance the field of image segmentation by enabling segmentation of unseen categories with minimal input.