Heavyweight Statistical Alignment to Guide Neural Translation.

Area of Science:

• Natural Language Processing
• Machine Translation
• Deep Learning

Background:

• Transformer models with multihead attention excel in translation.
• Prior alignment features from statistical models can benefit Transformer training.
• Lightweight prior alignment has shown effectiveness in guiding Transformer attention heads.

Purpose of the Study:

• To investigate the impact of applying heavyweight prior alignments to all attention heads in Transformer models.
• To enhance the performance of Transformer-based machine translation systems, particularly for low-resource languages.

Main Methods:

• Introduced a training cost incorporating alignment cost, defined as deviation from prior alignments, with a weight of 0.5.
• Averaged attention probabilities from all heads in the penultimate layer's multihead attention sublayer to increase the role of prior alignment.
• Evaluated the approach on an English-Vietnamese translation task.

Main Results:

• The proposed Transformer model achieved a BLEU score of 25.71, outperforming the baseline model (21.34) by 4.37 BLEU.
• Qualitative analysis through case studies by native speakers validated the improved translation quality.
• The approach demonstrated superior performance in training Transformer-based translation models.

Conclusions:

• Heavyweight prior alignments effectively guide all attention heads, leading to superior Transformer-based translation models.
• The proposed language-independent method is promising for improving machine translation across various language pairs, including Slavic languages.
• This work contributes valuable insights to the field of machine translation, particularly for under-resourced languages.