Orthogonal position representations for transformer in neural machine translation

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Machine Learning

Background:

• Transformer architecture dominates Neural Machine Translation (NMT) due to its ability to capture long-range dependencies.
• Self-attention mechanisms in Transformers are position-insensitive, necessitating effective positional encoding for sequence order.
• Existing positional encoding methods like sine-cosine and learnable embeddings have limitations.

Purpose of the Study:

• To introduce a novel orthogonal fixed-dimension positional representation (OPR) for Transformer-based NMT.
• To enhance the discrimination of token positions within sequences without adding parameters or computational overhead.
• To improve the performance of NMT models by providing superior sequential context.

Main Methods:

• Proposed a novel orthogonal fixed-dimension positional representation (OPR).
• Implemented OPR in Transformer models for NMT tasks.
• Conducted experiments on multiple standard NMT datasets.
• Evaluated performance using BLEU and COMET scores.

Main Results:

• OPR consistently outperformed baseline positional encoding methods (sine-cosine, learnable embeddings).
• Achieved significant improvements in BLEU and COMET scores across all tested language pairs.
• OPR demonstrated enhanced performance when combined with relative positional encoding (RPR).

Conclusions:

• OPR is an efficient and effective positional encoding method for NMT.
• OPR successfully models both absolute and relative positional relationships, crucial for sequence understanding.
• The proposed OPR offers a parameter-free and computationally inexpensive advancement for Transformer models.