SVE-Former:一个通过单向量嵌入的快速里埃变压器
Xiang-Jun Shen1, Wenxiu Tian1, Yang Yang1
1School of Computer Science and Communication Engineering, Jiangsu University, Jiangsu University Campus, Zhenjiang, 212013, Jiangsu, China.
概括
SVE-Former引入了单向量嵌入,通过捕获令牌依赖来提高变压器效率. 这种方法显著减少了培训时间,并提高了对远程依赖的性能.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 自然语言处理自然语言处理.
背景情况:
- 变压器中的自我注意力机制具有二次复杂性,导致高的培训成本.
- 现有的线性自我注意方法往往无法完全捕捉到代币间的依赖关系.
- 这限制了变压器在处理长序列的有效性.
研究的目的:
- 提出SVE-Former,一种变压器变体,可以提高自我注意力效率和表达力.
- 解决现有的线性自我注意方法在捕捉符号关系方面的局限性.
- 提高模型处理序列数据中长距离依赖性的能力.
主要方法:
- 通过对代币进行单数值分解 (SVD) 来提取数据子空间,采用单数向量嵌入 (SVE).
- 将注意力机制压缩成基于令牌子空间的紧表示.
- 介绍了福里埃域SVD计算效率的方法,使用稳定的福里埃基数和库尔巴克-莱布勒分歧来估计子空间稳定性.
主要成果:
- SVE-Former显著提高了表达力和计算效率.
- 里埃域SVD减少了训练所需的样本大小,并最大限度地减少了冗余的计算.
- 实验结果显示,SVE-Former的表现优于最先进的线性注意力机制,在基准数据集上,训练时间减少了10%以上.
结论:
- SVE-Former有效地捕捉了代币依赖性,改善了功能提取和信息保留的自我注意力.
- 拟议的方法为处理变压器中远程依赖提供了一个计算效率高和强大的替代方案.
- 与现有的线性注意力模型相比,SVE-Former表现出卓越的性能和效率.
相关概念视频
Fast Fourier Transform
854
The Fast Fourier Transform (FFT) is a computational algorithm designed to compute the Discrete Fourier Transform (DFT) efficiently. By breaking down the calculations into smaller, manageable sections, the FFT significantly reduces the computational complexity involved. Direct computation of an N-point DFT requires N2 complex multiplications, whereas the FFT algorithm needs only (N/2)log2N multiplications, offering a much faster performance.
The computational efficiency of the FFT becomes...
The computational efficiency of the FFT becomes...
854
Basic signals of Fourier Transform
862
The Fourier Transform is a pivotal mathematical tool in signal processing, enabling the transformation of time-domain signals into their frequency-domain representations. Among the numerous elements within this domain, certain functions like the sinc function, delta function, and exponential signals hold significant importance due to their unique properties and implications.
The sinc function, defined as sinc(x) = sin(πx)/(πx), is particularly notable for its symmetry and behavior at...
The sinc function, defined as sinc(x) = sin(πx)/(πx), is particularly notable for its symmetry and behavior at...
862
Cartesian Form for Vector Formulation
1.1K
The Cartesian form for vector formulation is a process to calculate the moment of force using the position and force vectors. The moment of force is defined as the cross-product of these vectors, making it a vector quantity. The Cartesian form of the position and force vectors involves unit vectors, which can be used to express the cross-product in determinant form.
1.1K
Vector Algebra: Method of Components
18.7K
It is cumbersome to find the magnitudes of vectors using the parallelogram rule or using the graphical method to perform mathematical operations like addition, subtraction, and multiplication. There are two ways to circumvent this algebraic complexity. One way is to draw the vectors to scale, as in navigation, and read approximate vector lengths and angles (directions) from the graphs. The other way is to use the method of components.
In many applications, the magnitudes and directions of...
In many applications, the magnitudes and directions of...
18.7K
Discrete Fourier Transform
827
The Discrete Fourier Transform (DFT) is a fundamental tool in signal processing, extending the discrete-time Fourier transform by evaluating discrete signals at uniformly spaced frequency intervals. This transformation converts a finite sequence of time-domain samples into frequency components, each representing complex sinusoids ordered by frequency. The DFT translates these sequences into the frequency domain, effectively indicating the magnitude and phase of each frequency component present...
827
Parseval's Theorem for Fourier transform
2.0K
Parseval's theorem is a fundamental principle in signal processing that enables the calculation of a signal's energy in either the time domain or the frequency domain. This theorem is pivotal in demonstrating energy conservation between these two domains, ensuring that the computed energy value remains consistent regardless of the domain of analysis.
To understand Parseval's theorem, it is essential to first comprehend how signal energy is typically calculated. When considering a...
To understand Parseval's theorem, it is essential to first comprehend how signal energy is typically calculated. When considering a...
2.0K


