Enhancing Perception of Key Changes in Remote Sensing Image Change Captioning.

Area of Science:

• Computer Vision
• Remote Sensing
• Artificial Intelligence

Background:

• Existing remote sensing image change captioning methods struggle to differentiate actual changes from irrelevant areas.
• This leads to models being influenced by extraneous features, reducing captioning accuracy.

Purpose of the Study:

• To develop a novel multimodal model for remote sensing image change captioning that accurately identifies and captions changes.
• To enhance the effectiveness and precision of change features by integrating pixel-level change detection.

Main Methods:

• Proposed the Key Change Features and Instruction-tuned (KCFI) model, incorporating a ViTs encoder, key feature perceiver, pixel-level change detection decoder, and an instruction-tuned large language model decoder.
• Employed a dynamic weight-averaging strategy for joint optimization of change captioning and detection tasks.
• Investigated optimal visual fine-tuning instructions, identifying key change features as the most effective guidance for the large language model.

Main Results:

• The KCFI model demonstrated superior performance compared to state-of-the-art methods on the LEVIR-CC dataset.
• The integration of pixel-level change detection effectively constrained key change features, improving captioning accuracy.
• Guiding the large language model with only key change features proved to be the optimal strategy.

Conclusions:

• The proposed KCFI model significantly advances remote sensing image change captioning by effectively filtering irrelevant information.
• The multimodal approach, combining change detection with instruction-tuned large language models, offers a promising direction for future research.
• The study provides a robust framework for accurate and efficient change captioning in remote sensing imagery.