Backpropagation-Based Decoding for Multimodal Machine Translation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Existing image-to-text models often require large, parallel datasets for cross-lingual transfer.
• Bridging the gap between visual understanding and diverse linguistic expressions remains a challenge.

Purpose of the Study:

• To develop a method for cross-lingual image translation without direct paired data.
• To leverage intermediate visual representations from deep convolutional neural networks for information transfer across languages.
• To enable translation between languages with limited or no parallel corpora.

Main Methods:

• Utilized backpropagation-based decoding.
• Employed transformer-based multilingual-multimodal language models.
• Trained on images associated with English, German, and Japanese text, without direct German-Japanese image pairs.

Main Results:

• Successfully translated German-Japanese and Japanese-German sentence pairs.
• Demonstrated general utility in multilingual image captioning tasks.
• Achieved favorable results on the Multi30k dataset compared to existing methods.

Conclusions:

• The proposed approach effectively transfers visual information across languages using intermediate representations.
• This method offers a viable solution for low-resource cross-lingual image understanding and translation.
• The technique shows promise for enhancing multilingual image captioning capabilities.