PET:在变压器上进行参数有效的知识蒸
Hyojin Jeon1, Seungcheol Park1, Jin-Gee Kim1
1Department of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea.
PloS one
|July 6, 2023
概括
本研究介绍了对变压器 (PET) 的参数高效知识蒸,这是一种创建更小,更快的变压器模型的方法. PET有效地压缩了编码器和解码器组件,大大提高了自然语言处理任务的效率.
科学领域:
- 自然语言处理自然语言处理.
- 机器学习 机器学习
- 人工智能的人工智能
背景情况:
- 变压器模型在NLP任务中提供了显著的性能提升.
- 大型变压器模型由于尺寸,计算成本和推断时间而存在部署挑战.
- 现有的压缩方法经常忽略解码器,这有助于推断延迟.
研究的目的:
- 开发一种高效的变压器压缩方法,减少编码器和解码器的尺寸.
- 为了保持压缩格式的大型变压器模型的性能.
- 为了使变压器模型能够在资源有限的设备上部署.
主要方法:
- 变压器 (PET) 上的参数高效知识蒸利用参数组之间的高效重量分配.
- 聚乙烯 (PET) 包含一个升温过程,简化了任务,以增强知识蒸收益.
- 该方法侧重于压缩变压器模型的编码器和解码器组件.
主要成果:
- 与未压缩模型相比,PET显著降低了内存使用量,并加快了推断速度.
- 在IWSLT'14 EN→DE机器翻译任务中,PET实现了81.20%的内存减少和45.15%的推断速度增加.
- 压缩导致BLEU分数略有下降,仅为0.27,表明性能保持.
结论:
- 对于压缩大型变压器模型,PET提供了一种有效的解决方案,以实现高效部署.
- 该方法优于现有的变压器压缩技术,特别是在机器翻译中.
- 聚乙烯可在没有显著性能下降的情况下实现显著的效率提升,使大型模型更容易获得.
相关概念视频
Transformers with Off-Nominal Turns Ratios
182
In scenarios involving parallel transformers with disparate ratings, developing per-unit models requires accommodating off-nominal turns ratios. This situation arises when the selected base voltages are not proportional to the transformer’s voltage ratings. Consider a transformer where the rated voltages are related by the term a. If the chosen voltage bases satisfy a relationship involving term b, term c is defined as the ratio of these bases. This ratio is then substituted into the...
182
Energy Losses in Transformers
907
In an ideal transformer, it is assumed that there are no energy losses, and, hence, all the power at the primary winding is transferred to the secondary winding. However, in reality, the transformers always have some energy losses, and, hence, the output power obtained at the secondary winding is less than the input power at the primary winding due to energy losses.
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
907
Transformers in Distribution System
127
Transformers in distribution systems can be broadly categorized into distribution substation transformers and other distribution transformers. They are crucial for stepping down high transmission voltages to levels suitable for distribution and end-user applications.
Distribution substation transformers come in various ratings and typically use mineral oil for insulation and cooling. To prevent moisture and air from entering the oil, some transformers use an inert gas like nitrogen to fill the...
Distribution substation transformers come in various ratings and typically use mineral oil for insulation and cooling. To prevent moisture and air from entering the oil, some transformers use an inert gas like nitrogen to fill the...
127
The Ideal Transformer
434
In single-phase two-winding transformers, two windings are coiled around a magnetic core characterized by cross-sectional area A and magnetic permeability μ. A phasor current i1 enters the left winding while i2 exits the right winding, establishing the fundamental working of the transformer through electromagnetic principles.
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
434
Types Of Transformers
1.0K
Transformers can provide desired voltages to a circuit by modifying the number of turns in the secondary windings.
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
1.0K
Three-Winding Transformers
269
Three identical single-phase transformers can be configured to form a three-phase transformer connection, which involves high-voltage and low-voltage windings. The high-voltage windings are denoted by capital letters A-B-C, while the low-voltage windings are labeled with lowercase letters a-b-c, representing their respective phases. This notation helps distinguish between the high and low voltage sides of the transformer.
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
In the per-unit equivalent circuit of a grounded Y-Y three-phase...
269


