在视觉变压器中的自我注意力矩阵的CUR分解
IEEE transactions on pattern analysis and machine intelligence
|December 19, 2025
概括
我们介绍CURSA,一种新的线性自我注意力方法,可以保持高性能,同时降低计算复杂性. 与现有方法相比,CURSA在各种视觉任务中实现了更好的数据效率,速度和准确性.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 变压器在NLP和计算机视觉方面非常成功.
- 自注意力机制是变压器的核心,但具有二次复杂性,限制了其在视觉中的使用.
- 现有的线性自我注意方法往往会为了降低复杂性而损害性能.
研究的目的:
- 提出一种关于自我注意的新型线性近似方法,称为CURSA.
- 在自我注意力机制中,同时实现高性能和低复杂性.
- 解决视觉任务中二次复杂性的局限性.
主要方法:
- 开发了CURSA,这是一种香草自我注意力机制的线性近似.
- 使用CUR分解来将大型矩阵乘法分解为较小的矩阵乘法.
- 通过这种分解实现了近乎线性复杂性.
主要成果:
- 与最先进的自我注意机制相比,CURSA表现出更高的性能.
- 图像分类,语义细分和对象检测方面的实验显示了显著的改进.
- CURSA实现了更高的数据效率,更快的处理速度和更高的准确性.
结论:
- 在视觉任务中,CURSA提供了一个有前途的解决方案,用于高效和高性能的自我注意力.
- 该方法有效地平衡了计算复杂性和模型准确性.
- 在各种基准测试中,CURSA的表现优于现有的线性注意力机制.
相关概念视频
Transformers
1.7K
A device that transforms voltages from one value to another using induction is called a transformer. A transformer consists of two separate coils, or windings, wrapped around the same soft iron core. However, they are electrically insulated from each other.
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
1.7K
Types Of Transformers
1.4K
Transformers can provide desired voltages to a circuit by modifying the number of turns in the secondary windings.
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
1.4K
Three-Winding Transformers
655
Three identical single-phase transformers can be configured to form a three-phase transformer connection, which involves high-voltage and low-voltage windings. The high-voltage windings are denoted by capital letters A-B-C, while the low-voltage windings are labeled with lowercase letters a-b-c, representing their respective phases. This notation helps distinguish between the high and low voltage sides of the transformer.
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
655
Inertia Tensor
1.1K
The concept of the inertia tensor is employed to depict the mass distribution and rotational inertia of a solid or rigid object. This tensor is expressed through a three-by-three matrix. Each component within this matrix corresponds to varying moments of inertia about specific axes.
The diagonal components of the inertia tensor matrix represent the moments of inertia concerning the principal axes of the object. These primary axes are defined as the axes where the object experiences the least...
The diagonal components of the inertia tensor matrix represent the moments of inertia concerning the principal axes of the object. These primary axes are defined as the axes where the object experiences the least...
1.1K
Transformers with Off-Nominal Turns Ratios
491
In scenarios involving parallel transformers with disparate ratings, developing per-unit models requires accommodating off-nominal turns ratios. This situation arises when the selected base voltages are not proportional to the transformer’s voltage ratings. Consider a transformer where the rated voltages are related by the term a. If the chosen voltage bases satisfy a relationship involving term b, term c is defined as the ratio of these bases. This ratio is then substituted into the...
491
Energy Losses in Transformers
1.3K
In an ideal transformer, it is assumed that there are no energy losses, and, hence, all the power at the primary winding is transferred to the secondary winding. However, in reality, the transformers always have some energy losses, and, hence, the output power obtained at the secondary winding is less than the input power at the primary winding due to energy losses.
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
1.3K

