Cross-Modal Remote Sensing Image-Text Retrieval via Context and Uncertainty-Aware Prompt

Area of Science:

• Remote Sensing
• Computer Vision
• Artificial Intelligence

Background:

• Cross-modal remote sensing image-text retrieval (CMRSITR) is a growing field, leveraging large pretrained models.
• Existing methods face challenges with high computational costs for fine-tuning and domain gaps between natural and remote sensing images.
• These limitations hinder the effective application of powerful image-text models in remote sensing contexts.

Purpose of the Study:

• To propose a novel CMRSITR network, Context and Uncertainty-aware Prompt (CUP), addressing computational and domain adaptation challenges.
• To enable efficient knowledge transfer from large pretrained models to remote sensing tasks using prompt tuning.
• To enhance the model's understanding of remote sensing image characteristics and mitigate uncertainties for improved retrieval accuracy.

Main Methods:

• Implemented prompt tuning to reduce computational burden by training only prompt tokens.
• Developed a Prompt Generation Module (PGM) to create remote sensing-specific prompt tokens, bridging the gap with natural image pretraining.
• Integrated an Uncertainty Estimation Module (UEM) to address semantic misalignment and data uncertainties.

Main Results:

• The proposed CUP network achieved competitive performance on three benchmark datasets for CMRSITR.
• Prompt tuning significantly reduced the need for extensive computational resources compared to full model fine-tuning.
• The RS-oriented prompts and uncertainty estimation effectively improved the model's ability to interpret remote sensing imagery.

Conclusions:

• CUP offers an efficient and effective solution for cross-modal remote sensing image-text retrieval.
• The method successfully adapts large-scale models to the unique characteristics of remote sensing data.
• The approach demonstrates strong potential for advancing remote sensing information extraction and analysis.