AdaAlign: A unified solution for traditional and modern zero-shot sketch-based image retrieval

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Zero-shot sketch-based image retrieval (ZS-SBIR) faces challenges due to domain differences between sketches and photos, and the semantic gap between known and unknown classes.
• Traditional methods using small vision encoders and modern large vision-language models (VLMs) have limitations, with no unified approach effectively addressing ZS-SBIR challenges.

Purpose of the Study:

• To propose an effective "Adaptation and Alignment (AdaAlign)" approach to overcome the limitations of existing methods in ZS-SBIR.
• To improve cross-domain representation and bridge the semantic gap for more generalizable visual representations.

Main Methods:

• Implemented lightweight Adapters or LoRA to learn abstract sketch concepts and enhance cross-domain representation, alleviating domain heterogeneity.
• Proposed a distillation framework to align learned image embeddings with semantically richer text embeddings, bridging the semantic gap.
• Integrated AdaAlign into both traditional small models (ResNet50, DINO-S) and modern VLMs (SigLIP).

Main Results:

• Achieved state-of-the-art performance by integrating AdaAlign into various models.
• Demonstrated the method's superiority in retrieval accuracy and flexibility through extensive experiments on three benchmark datasets.

Conclusions:

• The AdaAlign approach effectively addresses key challenges in ZS-SBIR by improving cross-domain capabilities and semantic alignment.
• The proposed method offers a unified and flexible solution, achieving superior performance across different model architectures.