Augmented Multimodality Fusion for Generalized Zero-Shot Sketch-Based Visual Retrieval.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Zero-shot sketch-based image retrieval (ZS-SBIR) shows promise for real-world applications.
• Existing ZS-SBIR methods struggle with scenarios involving both seen and unseen categories.
• Generalized zero-shot sketch-based image retrieval (GZS-SBIR) addresses this gap.

Purpose of the Study:

• To propose a novel framework for generalized zero-shot sketch-based image retrieval (GZS-SBIR).
• To effectively generalize seen concepts to unobserved categories in sketch-based image retrieval.
• To handle practical scenarios where query sketches and gallery photos contain mixed categories.

Main Methods:

• Developed an Augmented Multi-modality Fusion (AMF) framework.
• Introduced a cross-domain augmentation module in visual and semantic spaces to mimic novel knowledge.
• Implemented a triplet domain alignment module for coupling cross-domain distributions and embedding propagation for feature refinement.

Main Results:

• The proposed AMF framework demonstrated superior performance on popular ZS-SBIR benchmarks.
• Evaluations using a new GZS-SBIR protocol on the DomainNet dataset confirmed the method's effectiveness.
• The approach successfully generalized concepts to unseen categories, outperforming baseline methods.

Conclusions:

• The AMF framework provides an efficient solution for the GZS-SBIR challenge.
• Cross-domain augmentation and triplet domain alignment are key to generalizing concepts.
• The proposed method offers a robust and effective approach for practical sketch-based image retrieval systems.