持续的低等级缩放的点-产品注意力
Ginés Carreto Picón1, Illia Oleksiienko1, Lukas Hedegaard1
1Department of Electrical and Computer Engineering, Aarhus University, Finlandsgade 22, Aarhus N, 8200, Denmark.
概括
这项研究引入了对Scaled Dot-product Attention的Nyström近似,显著降低了流数据处理的计算成本. 这种新方法可以在变压器中实现高效的连续推断,而不会牺牲性能.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 变压器擅长对静态任务进行序列处理.
- 变压器中的Scaled Dot产品Attention具有较高的计算和内存需求,限制了其在实时数据流处理中的使用.
- 降低变压器成本的现有方法包括低级近似和稀疏性.
研究的目的:
- 开发一个高效的调度点产品注意力配方,适合连续推理.
- 解决变压器在数据流应用中的计算和内存限制.
主要方法:
- 引入了使用尼斯特罗姆近似的缩放点产物注意力的新配方.
- 开发了一种使用变压器进行高效的连续推理的方法.
主要成果:
- 与原来的变压器相比,提议的连续缩放点产品注意力减少了多达三倍的操作.
- 保持了与在线任务中的现有模型可比的预测性能.
结论:
- 尼斯特罗姆近似为Scaled Dot-product Attention在流数据场景中提供了一个计算效率高的替代方案.
- 这种方法使得变压器更适合实时应用,资源有限.
相关概念视频
Dot Product
760
The dot product is an essential concept in mathematics and physics.
In engineering, the dot product of any two vectors is the product of the magnitudes of the vectors and the cosine of the angle between them. It is denoted by a dot symbol between the two vectors.
Consider a vehicle pulling an object along the ground using a rope. If the rope makes an angle with the horizontal axis, the work done can be calculated using the dot product of the force applied and the object's displacement.
The dot...
In engineering, the dot product of any two vectors is the product of the magnitudes of the vectors and the cosine of the angle between them. It is denoted by a dot symbol between the two vectors.
Consider a vehicle pulling an object along the ground using a rope. If the rope makes an angle with the horizontal axis, the work done can be calculated using the dot product of the force applied and the object's displacement.
The dot...
760
The Dot Product
149
Measuring how one directional quantity affects another along a specific path involves comparing their orientation and strength. When two such quantities are represented using direction and amount, a numerical result is computed to show how much one acts along the path of the other. This result comes from a rule combining both inputs' horizontal and vertical parts and adding the results.This calculation gives a single value that grows larger when both inputs point in similar directions and...
149
Dot Product: Problem Solving
655
The dot product is a powerful tool in problem-solving involving vectors, given that the dot product of two vectors is the product of their magnitudes and the cosine of the angle between them measured anti-clockwise. Solving problems involving the dot product requires understanding its properties and developing a step-by-step process to solve them. Here are the main steps to follow when solving any general problem involving the dot product:
Identify the problem: Start by reading the problem and...
Identify the problem: Start by reading the problem and...
655
Scalar Product (Dot Product)
25.9K
The scalar multiplication of two vectors is known as the scalar or dot product. As the name indicates, the scalar product of two vectors results in a number, that is, a scalar quantity. Scalar products are used to define work and energy relations. For example, the work that a force (a vector) performs on an object while causing its displacement (a vector) is defined as a scalar product of the force vector with the displacement vector.
The scalar product of two vectors is obtained by multiplying...
The scalar product of two vectors is obtained by multiplying...
25.9K
Reducing Line Loss
338
In a three-phase circuit, line loss is an indicator of energy dissipated as heat due to the resistance of transmission lines. To address this, incorporating transformers into the system—a step-up transformer at the source and a step-down transformer at the load—is a strategic solution. Two three-phase transformers are introduced to improve this.
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss in...
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss in...
338
Mean Absolute Deviation
3.2K
The mean absolute deviation is also a measure of the variability of data in a sample. It is the absolute value of the average difference between the data values and the mean.
Let us consider a dataset containing the number of unsold cupcakes in five shops: 10, 15, 8, 7, and 10. Initially, calculate the sample mean. Then calculate the deviation, or the difference, between each data value and the mean. Next, the absolute values of these deviations are added and divided by the sample size to...
Let us consider a dataset containing the number of unsold cupcakes in five shops: 10, 15, 8, 7, and 10. Initially, calculate the sample mean. Then calculate the deviation, or the difference, between each data value and the mean. Next, the absolute values of these deviations are added and divided by the sample size to...
3.2K

