Sketch-Segformer: Transformer-Based Segmentation for Figurative and Creative Sketches

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Sketch semantic segmentation is crucial for understanding sketches.
• Existing methods often focus on limited aspects of sketch data (e.g., whole image, strokes, or sequences).
• There's a need to explore complementary information across multiple facets of sketch data.

Purpose of the Study:

• To propose a novel framework, Sketch-Segformer, for sketch semantic segmentation.
• To demonstrate the benefit of integrating multi-facet sketch information.
• To achieve state-of-the-art performance on both traditional and creative sketches.

Main Methods:

• Developed a transformer-based framework (Sketch-Segformer) that treats sketches as stroke sequences.
• Introduced two self-attention modules with different receptive fields (whole sketch and individual stroke).
• Integrated order, spatial, and stroke-level embeddings.

Main Results:

• Achieved state-of-the-art performance on figurative sketch datasets (SPG, SketchSeg-150K).
• Demonstrated strong performance on creative sketches (CreativeSketch dataset) by utilizing multi-facet information.
• Ablation studies, visualizations, and invariance tests validated the design choices.

Conclusions:

• The proposed Sketch-Segformer effectively utilizes complementary multi-facet sketch information.
• The framework offers a robust approach for sketch semantic segmentation across various sketch types.
• The study highlights the importance of holistic sketch data exploration for improved interpretation.