Read, spot and translate.

Area of Science:

• Natural Language Processing
• Computer Vision
• Machine Learning

Background:

• Multimodal machine translation (MMT) aims to enhance translation by incorporating visual information.
• Existing MMT models often use whole images or abstract regions for visual grounding, which can be suboptimal.

Purpose of the Study:

• To propose and evaluate novel multimodal machine translation (MMT) approaches that leverage object-level referential grounding.
• To investigate two distinct referential grounding strategies: implicit and explicit grounding.

Main Methods:

• Developed MMT models that ground source language words to specific image objects.
• Implemented implicit grounding, where visual grounding and translation are learned jointly.
• Implemented explicit grounding, where visual grounding is performed independently and then used to guide translation.
• Conducted experiments on the Multi30K dataset for English-German, English-French, and English-Czech language pairs.

Main Results:

• Referential grounding models focusing on objects demonstrated superior performance over existing MMT approaches.
• Both implicit and explicit grounding methods showed significant improvements in translation quality.
• Evaluations using automatic metrics and human judgment confirmed the effectiveness of the proposed methods.

Conclusions:

• Object-level referential grounding in the source language is a more effective strategy for MMT than previous methods.
• The proposed implicit and explicit grounding approaches offer viable pathways for advancing MMT systems.
• This work highlights the importance of fine-grained visual grounding for improving cross-modal translation.