Zero-shot sketch-based remote sensing image retrieval based on cross-modal fusion

Area of Science:

• Computer Science
• Remote Sensing
• Artificial Intelligence

Background:

• Sketch-based image retrieval (SBIR) enables users to find images using hand-drawn sketches.
• Zero-shot SBIR (ZS-SBIR) is crucial when no example images are available.
• ZS-SBIR for remote sensing data (ZS-SBRSIR) faces challenges due to scarce sketch data and a significant domain gap.

Purpose of the Study:

• To address the limitations in current zero-shot sketch-based remote sensing image retrieval (ZS-SBRSIR).
• To develop a novel network that effectively bridges the domain gap between sketches and remote sensing images.
• To improve the generalization ability of ZS-SBRSIR models.

Main Methods:

• Introduced a novel zero-shot cross-modal fusion network for ZS-SBRSIR.
• Extracted edge features from remote sensing images as a bridge modality, closer to sketches.
• Utilized existing image labels for contrastive training to incorporate semantic information.

Main Results:

• The proposed model effectively leverages multi-modal information to bridge the domain gap.
• Edge features from remote sensing images enhance the fusion process and improve zero-shot generalization.
• Comprehensive experiments validated the model's efficacy on ZS-SBRSIR tasks.

Conclusions:

• The novel cross-modal fusion network significantly advances ZS-SBRSIR capabilities.
• Leveraging readily available multi-modal data, such as edge features and semantic labels, is key to overcoming ZS-SBRSIR challenges.
• The approach demonstrates improved performance and generalization for sketch-based retrieval in remote sensing.