Feature Fusion and Metric Learning Network for Zero-Shot Sketch-Based Image Retrieval

Area of Science:

• Computer Vision
• Machine Learning

Background:

• Zero-shot sketch-based image retrieval (ZS-SBIR) faces challenges due to the abstract nature of sketches and the domain gap between sketches and real-world photos.
• Standard backbone networks struggle to capture semantic similarities between sketches and photos without textual aids.

Purpose of the Study:

• To propose a novel feature embedding model, Attention Map Feature Fusion (AMFF), for effective ZS-SBIR.
• To address the domain gap by introducing a domain-aware triplets (DAT) optimization method.

Main Methods:

• The AMFF model integrates ResNet-50 for feature extraction and an attention network for enhanced representation.
• Attention maps are generated by processing ResNet-50 residuals, avoiding external semantic knowledge.
• Domain-aware triplets (DAT) are employed to learn domain feature discrimination and semantic feature embedding, optimizing the network.

Main Results:

• The proposed method demonstrates superior performance compared to state-of-the-art approaches.
• On the Tu-berlin dataset, an accuracy of 61.2 + 1.2% Prec200 was achieved.
• On the Sketchy_c100 dataset, results included 62.3 + 3.3% mAPall and 75.5 + 1.5% Prec100.

Conclusions:

• The AMFF model combined with DAT effectively tackles the ZS-SBIR problem by bridging the domain gap.
• The approach achieves state-of-the-art results, highlighting its efficacy in cross-domain image retrieval.