C³aptioner: Improving change captioning by leveraging momentum cross-view and cross-modality contrastive learning

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Change captioning aims to describe visual differences between images.
• Existing methods focus on detecting and describing changes, neglecting persistent elements and vision-language links.
• Effective change captioning requires describing both changes and unchanged semantic elements, linking visual cues to language.

Purpose of the Study:

• To develop an advanced change captioning model, C³aptioner, that addresses limitations of prior work.
• To enhance descriptions by incorporating both changed and unchanged visual information.
• To establish a robust connection between visual features and linguistic descriptions.

Main Methods:

• The C³aptioner model utilizes dual momentum contrastive learning objectives.
• It employs intra-image and inter-image Transformer encoders for visual feature extraction.
• Cross-view and cross-modality contrastive learning objectives align visual and textual representations, addressing viewpoint variations and modality gaps.

Main Results:

• The C³aptioner model achieves state-of-the-art performance across five datasets.
• Significant improvements were observed in challenging scenarios with extreme viewpoint changes.
• The dual contrastive approach effectively models both changed and unchanged elements, enhancing vision-language correspondence.

Conclusions:

• C³aptioner provides more contextually rich and human-like change descriptions.
• The proposed dual contrastive learning strategy is effective for change captioning.
• The model demonstrates superior performance, particularly in handling viewpoint variations.