ReCoTR: Reducing Semantic Cognitive Shift via Dual-Consensus Token Compression for Remote Sensing Image-Text...

Area of Science:

• Remote Sensing
• Computer Vision
• Artificial Intelligence

Background:

• Vision-language models (VLMs) show promise for remote sensing (RS) data analysis.
• Semantic shift in RS images challenges VLM transferability and performance.
• Existing methods struggle with region-level bias and background noise in RS data.

Purpose of the Study:

• To develop an enhanced CLIP-based framework, ReCoTR, for cross-modal retrieval in remote sensing.
• To address semantic shift, region-level granularity bias, and contextual semantic drift in RS imagery.
• To improve the accuracy and robustness of semantic understanding for large-scale RS datasets.

Main Methods:

• Proposed ReCoTR framework utilizing a Dual Consensus Token Evaluation (DCTE) module.
• DCTE employs a mixture-of-experts strategy to fuse inter-modal semantic consensus and intra-modal structural consistency.
• Introduced Semantic Confidence Token Compression (SCTC) module to filter and aggregate semantically relevant tokens, reducing noise.

Main Results:

• ReCoTR demonstrated superior performance on bidirectional image-text retrieval tasks across three benchmark RS datasets.
• The framework effectively mitigates region-level granularity bias and contextual semantic drift.
• ReCoTR shows robustness in handling background noise and improving semantic alignment in RS data.

Conclusions:

• ReCoTR significantly enhances cross-modal retrieval for remote sensing data.
• The proposed DCTE and SCTC modules effectively address key challenges in RS image-text understanding.
• ReCoTR offers a robust solution for applications in urban governance, environmental monitoring, and disaster response.