Aligning Source Visual and Target Language Domains for Unpaired Video Captioning.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Supervised video captioning requires paired video-caption data, which is scarce for many languages.
• Existing pipeline methods for unpaired video captioning suffer from visual irrelevance and error propagation.

Purpose of the Study:

• To develop a novel system for unpaired video captioning that overcomes the limitations of pipeline approaches.
• To improve the quality and visual relevance of generated captions in low-resource languages.

Main Methods:

• Proposed the Unpaired Video Captioning with Visual Injection (UVC-VI) system.
• Introduced a Visual Injection Module (VIM) to align visual and textual domains, injecting visual information directly.
• Developed a Multimodal Collaborative Encoder (MCE) to enhance cross-modality information transfer.

Main Results:

• UVC-VI significantly outperforms traditional pipeline systems in generating video captions.
• The proposed system surpasses several existing supervised video captioning models.
• Integrating the Multimodal Collaborative Encoder (MCE) into supervised systems improved state-of-the-art performance by 4% and 7% on MSVD and MSR-VTT datasets.

Conclusions:

• UVC-VI offers an effective solution for unpaired video captioning, addressing visual irrelevance and error propagation.
• The Visual Injection Module (VIM) and Multimodal Collaborative Encoder (MCE) are key innovations for enhancing cross-modal understanding.
• The approach demonstrates the potential for improving video captioning in low-resource scenarios and advancing supervised methods.