Difference-Aware Multi-Grained Feature Learning With Memory-Attended Joint Interaction Transformer for Image Difference Captioning

Image Difference Captioning (IDC) aims to describe semantic changes between a pair of similar images using natural language. Most existing approaches rely on off-the-shelf visual backbones to extract features from image pairs. However, these backbones, pre-trained for single-image understanding, often struggle to capture nuances between two images. As a result, the extracted features are less effective for IDC. In this paper, we propose a two-stage training paradigm for IDC. In the first stage, we design a difference-aware multi-grained feature learning (DAMFEL) to adapt a pre-trained CLIP to IDC by jointly modeling coarse- and fine-grained contrastive alignments between image pairs and captions. This paradigm enables the model to sufficiently capture difference-aware features for image pairs. In the second stage, based on these features, we devise a memory-attended joint interaction transformer (MAJIT) for caption generation. MAJIT establishes mutual synergy between (i) cross-image interaction for perceiving visual changes and (ii) cross-modal interaction for grounding these changes to the generated words. At each decoding step, MAJIT employs learnable triplet shared memories to record the shared information among the two images and the partially generated caption, enabling the outcomes of both interactions to complement and reinforce each other. Experiments on four public IDC benchmarks demonstrate strong performance across diverse change scenarios, with consistent advantages over lightweight methods and competitive results relative to MLLM-based approaches.