Scale-Aware Prompting With Optimal Transport for Remote Sensing Image Captioning

Area of Science:

• Computer Vision
• Artificial Intelligence
• Remote Sensing

Background:

• Remote sensing image captioning is crucial for understanding complex scenes.
• Accurately describing objects, their attributes, and dependencies in these images is challenging.
• Existing methods struggle with fine-grained understanding and cross-modal alignment.

Purpose of the Study:

• To propose a novel method for accurate remote sensing image captioning.
• To enhance the representation of object attributes and dependencies in complex scenes.
• To improve cross-modal alignment between image features and textual descriptions.

Main Methods:

• Developed a Scale-aware Prompting with Optimal Transport (SPOT) model.
• Utilized a scale-aware prompt extractor to query multi-scale features and embed positional relations.
• Implemented fine-grained cross-modal alignment using optimal transport.
• Employed a caption Transformer with causal self-attention for caption generation.

Main Results:

• The proposed SPOT method achieves state-of-the-art performance on three public datasets.
• Ablation studies confirm the effectiveness of each component of the SPOT model.
• The method demonstrates superior capability in generating accurate captions for diverse remote sensing scenes.

Conclusions:

• The SPOT method significantly advances remote sensing image captioning.
• The integration of scale-aware prompting and optimal transport is effective for cross-modal alignment.
• This approach provides a robust framework for fine-grained understanding of remote sensing imagery.