Video Pivoting Unsupervised Multi-Modal Machine Translation.

Area of Science:

• Natural Language Processing
• Computer Vision
• Machine Learning

Background:

• Unsupervised machine translation (UMT) struggles with aligning source-target sentences in latent space.
• Unsupervised multi-modal machine translation (UMMT) leverages visual content to improve UMT by facilitating alignment.
• Existing UMMT models often rely on images and overlook the relational and temporal dynamics present in videos, leading to sensitivity to spurious correlations.

Purpose of the Study:

• To enhance unsupervised machine translation (UMT) by effectively utilizing video content.
• To address the limitations of current UMMT models in capturing object relations and temporal information.
• To improve latent space alignment in UMMT through explicit modeling of object interactions.

Main Methods:

• Employed a spatial-temporal graph derived from videos to capture object interactions across space and time.
• Developed a multi-modal back-translation framework incorporating pseudo-visual pivoting.
• Learned a shared multilingual visual-semantic embedding space and utilized visually pivoted captioning for weak supervision.

Main Results:

• The proposed model demonstrates validated translation capabilities on sentence-level and word-level tasks.
• The approach effectively disambiguates through object interaction modeling, promoting latent space alignment.
• The model shows strong generalization, performing well even when videos are absent during testing.

Conclusions:

• Spatial-temporal graphs from videos offer a powerful way to model object interactions for UMMT.
• The integration of multi-modal back-translation and pseudo-visual pivoting significantly advances UMMT.
• This research provides a robust UMMT solution that is less susceptible to spurious correlations and performs well across various conditions.