AMPLIFY:以注意力为基础的混合,用于提高变压器的性能和标签光滑
1School of Information Science and Technology, Yunnan Normal University, Kunming, Yunnan, China.
PeerJ. Computer science
|June 10, 2024
概括
AMPLIFY是一种新的数据增强技术,通过利用变压器注意力机制来降低模型对噪音数据的敏感性. 这种方法提高了文本分类性能,计算成本低,性能优于现有的方法.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 自然语言处理自然语言处理.
背景情况:
- 混合是一个数据增强技术,它结合样本以改善模型概括性.
- 标准混合方法可以将噪声和异常值从原始样本传播到增强数据中,从而对模型性能产生负面影响.
- 现有的混合技术可能需要大量的计算资源.
研究的目的:
- 引入AMPLIFY,一种新的混合方法,旨在减轻噪声和异常值的传播.
- 增强数据训练模型的强度和性能.
- 为了在低计算开销的情况下实现这些改进.
主要方法:
- AMPLIFY利用了变压器固有的注意力机制,可以选择性地减少噪音或异常特征的影响.
- 该方法可以与现有的变压器架构无集成,而无需添加可训练的参数.
- 与其他先进的混合策略相比,它运行时的计算成本最小.
主要成果:
- 在七个基准数据集中,AMPLIFY在文本分类任务中表现出卓越的表现.
- 提出的方法有效地减少了模型对异常值和噪音样本的过度敏感性.
- 在显著降低计算资源消耗的同时取得了最先进的结果.
结论:
- 在文本分类中,AMPLIFY提供了一种有效且计算效率高的解决方案,用于噪声强大的数据增强.
- 注意力机制的整合提供了一种新的方法来增强预训练模型,如BERT,ALBERT,RoBERTa和GPT.
- 这项研究为通过智能数据增强来提高深度学习模型性能开辟了新的途径.
更多相关视频
13:00Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments
Published on: January 23, 2017
9.9K
08:45Mapping Cortical Dynamics Using Simultaneous MEG/EEG and Anatomically-constrained Minimum-norm Estimates: an Auditory Attention Example
Published on: October 24, 2012
14.7K
相关概念视频
Energy Losses in Transformers
862
In an ideal transformer, it is assumed that there are no energy losses, and, hence, all the power at the primary winding is transferred to the secondary winding. However, in reality, the transformers always have some energy losses, and, hence, the output power obtained at the secondary winding is less than the input power at the primary winding due to energy losses.
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
862
Types Of Transformers
968
Transformers can provide desired voltages to a circuit by modifying the number of turns in the secondary windings.
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
968
Transformers with Off-Nominal Turns Ratios
149
In scenarios involving parallel transformers with disparate ratings, developing per-unit models requires accommodating off-nominal turns ratios. This situation arises when the selected base voltages are not proportional to the transformer’s voltage ratings. Consider a transformer where the rated voltages are related by the term a. If the chosen voltage bases satisfy a relationship involving term b, term c is defined as the ratio of these bases. This ratio is then substituted into the...
149
Three-Winding Transformers
222
Three identical single-phase transformers can be configured to form a three-phase transformer connection, which involves high-voltage and low-voltage windings. The high-voltage windings are denoted by capital letters A-B-C, while the low-voltage windings are labeled with lowercase letters a-b-c, representing their respective phases. This notation helps distinguish between the high and low voltage sides of the transformer.
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
222
Transformers
1.1K
A device that transforms voltages from one value to another using induction is called a transformer. A transformer consists of two separate coils, or windings, wrapped around the same soft iron core. However, they are electrically insulated from each other.
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
1.1K
Equivalent Circuits for Practical Transformers
413
The practical equivalent circuits of single-phase two-winding transformers exhibit significant deviations from their idealized versions due to the inherent properties of winding resistance and finite core permeability. These properties result in real and reactive power losses, affecting the transformer's performance. Understanding these deviations is crucial for designing more efficient transformers.
In a practical transformer, each winding exhibits resistance and leakage reactance. The...
In a practical transformer, each winding exhibits resistance and leakage reactance. The...
413
