On the Limitations and Capabilities of Position Embeddings for Length Generalization.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Position Embeddings (PEs) are crucial for Transformer models, impacting Length Generalization (LG).
• The fundamental role and limitations of PEs in achieving LG remain unclear.
• Existing PEs face challenges in adapting to varying computational demands and positional roles across different scales.

Purpose of the Study:

• To identify and formalize the fundamental limitations of Position Embeddings in achieving Length Generalization.
• To introduce theoretical frameworks and practical strategies for enhancing LG in Transformer models.
• To propose novel approaches for improving the adaptability and scalability of PEs.

Main Methods:

• Introduction of 'computational representation complexity' to measure operator requirements.
• Formalization of 'canonicality' to assess operator-position consistency across scales.
• Development of a learning-based position embedding framework with 'scale hint' for improved LG.

Main Results:

• PEs are proven unable to achieve LG for mappings with increasing computational complexity or non-canonical structures.
• LG is achievable with PEs when these limitations are absent, using PEs that characterize the computational representation.
• The proposed 'scale hint' and learning-based framework enhance LG for non-canonical mappings.

Conclusions:

• Transformer Position Embeddings have inherent limitations hindering Length Generalization.
• Computational representation complexity and canonicality are key factors determining PE performance for LG.
• Novel methods like 'scale hint' and learning-based PEs offer practical solutions for improving Transformer LG.