SeqPE: Transformer with Sequential Position Encoding

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Transformers lack inherent spatial understanding due to permutation-invariant self-attention layers.
• Traditional position embeddings (PEs) have limited extrapolation beyond trained sequence lengths.
• Existing advanced methods (ALIBI, ROPE) require significant adaptation for new modalities.

Purpose of the Study:

• To present SEQPE, a unified, fully learnable position encoding framework.
• To improve adaptability and scalability of positional encodings across diverse applications.
• To enhance extrapolation performance in Transformers.

Main Methods:

• SEQPE represents n-dimensional position indices as symbolic sequences.
• A lightweight sequential encoder learns position embeddings end-to-end.
• Regularization uses a contrastive objective and knowledge distillation for improved extrapolation.

Main Results:

• SEQPE surpasses strong baselines in language modeling, QA, and image classification.
• Significant improvements observed in perplexity, exact match (EM), and accuracy, especially during context length extrapolation.
• Demonstrates seamless generalization to multi-dimensional inputs without architectural redesign.

Conclusions:

• SEQPE offers a flexible and effective solution for position encoding in Transformers.
• The framework enhances model performance and generalization capabilities.
• SEQPE addresses key limitations in adaptability and scalability for positional information.