Cross-Domain Image Captioning via Cross-Modal Retrieval and Model Adaptation

Area of Science:

• Computer Vision
• Natural Language Processing
• Machine Learning

Background:

• Large-scale paired image-sentence datasets fuel image captioning success.
• Collecting domain-specific paired data is resource-intensive.
• Transfer learning offers a solution for adapting models to new domains with limited data.

Purpose of the Study:

• To develop an effective method for cross-domain image captioning using unpaired data.
• To leverage cross-modal retrieval for generating pseudo image-sentence pairs in target domains.
• To adapt image captioning models trained on source domains to new target domains.

Main Methods:

• Propose a cross-modal retrieval aided approach for cross-domain image captioning.
• Utilize an iterative cross-modal retrieval process to generate and refine pseudo image-sentence pairs.
• Employ an adaptive image captioning model with self-attention, fine-tuned on pseudo pairs.

Main Results:

• The proposed method achieves competitive or superior performance compared to state-of-the-art approaches across multiple target domains.
• Demonstrated effectiveness in cross-domain image captioning tasks using MSCOCO as the source domain.
• Successfully extended to cross-domain video captioning, validating the method's versatility.

Conclusions:

• The cross-modal retrieval approach effectively addresses the challenge of limited paired data in cross-domain image captioning.
• Iterative refinement of pseudo pairs enhances model adaptation and captioning quality.
• The method shows promise for various cross-modal generation tasks beyond image captioning.