SVE-Former: A fast fourier transformer via singular vector embedding.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Self-attention mechanisms in transformers have quadratic complexity, leading to high training costs.
• Existing linear self-attention methods often fail to fully capture inter-token dependencies.
• This limits the effectiveness of transformers in processing long sequences.

Purpose of the Study:

• To propose SVE-Former, a transformer variant that enhances self-attention efficiency and expressive power.
• To address the limitations of existing linear self-attention methods in capturing token relationships.
• To improve the model's ability to handle long-range dependencies in sequence data.

Main Methods:

• Employs singular vector embedding (SVE) by performing singular value decomposition (SVD) on tokens to extract data subspaces.
• Compresses the attention mechanism into a compact representation based on the token subspace.
• Introduces a Fourier-domain SVD method for computational efficiency, using stable Fourier bases and Kullback-Leibler divergence for subspace stability estimation.

Main Results:

• SVE-Former significantly enhances expressive power and computational efficiency.
• The Fourier-domain SVD reduces the required sample size for training and minimizes redundant computations.
• Experimental results show SVE-Former outperforms state-of-the-art linear attention mechanisms, achieving over 10% reduction in training time on benchmark datasets.

Conclusions:

• SVE-Former effectively captures token dependencies, improving self-attention for feature extraction and information retention.
• The proposed method offers a computationally efficient and powerful alternative for processing long-range dependencies in transformers.
• SVE-Former demonstrates superior performance and efficiency compared to existing linear attention models.