通过构成进行分布外概括:在变压器中,透镜通过感应头
Jiajun Song1, Zhuoyan Xu2, Yiqiao Zhong2
1State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence, Beijing 100080, China.
概括
大型语言模型 (LLM) 通过构建自我注意层来实现分布外 (OOD) 泛化. 一个共享的隐藏子空间,共同的桥梁表示,使得这个规则推断和新任务的组成.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 自然语言处理自然语言处理.
背景情况:
- 大型语言模型 (LLM) 显示出新兴的能力,包括用很少的例子解决新任务.
- 分布外 (OOD) 泛化,即在未见的数据分布上执行的能力,对于LLM功能至关重要.
- 在LLM OOD泛化背后的机制,特别是在规则推理任务中,尚不清楚.
研究的目的:
- 调查大型语言模型如何实现分布外通用化.
- 探索隐藏规则推断和构成在新任务LLM表现中的作用.
- 在OOD泛化过程中检查变压器模型,特别是感应头的内部动态.
主要方法:
- 在合成数据上实证检查变压器训练动态.
- 对各种预训练的LLM进行了广泛的实验,重点是自我注意力机制.
- 在符号推理的上下文学习环境中分析规则推理.
主要成果:
- 分布之外的概括与LLM中的作曲能力密切相关.
- 模型可以通过组成两个自我注意层来学习隐藏的规则,从而导致OOD概括.
- 一个共享的潜子空间,称为共同的桥梁表示,通过对准早期和后期层来促进构成.
结论:
- 通过自我注意层的组成,LLM实现了OOD泛化.
- 常见的桥梁表示假设解释了潜在特征对齐如何使规则组成成为可能.
- 了解这些机制是解锁更强大,更可通用的AI系统的关键.
更多相关视频
09:38Fabrication Procedures and Birefringence Measurements for Designing Magnetically Responsive Lanthanide Ion Chelating Phospholipid Assemblies
Published on: January 3, 2018
08:23A Random-displacement Measurement by Combining a Magnetic Scale and Two Fiber Bragg Gratings
Published on: September 30, 2019
相关概念视频
Transformers
1.0K
A device that transforms voltages from one value to another using induction is called a transformer. A transformer consists of two separate coils, or windings, wrapped around the same soft iron core. However, they are electrically insulated from each other.
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
1.0K
Types Of Transformers
943
Transformers can provide desired voltages to a circuit by modifying the number of turns in the secondary windings.
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
943
Energy Losses in Transformers
820
In an ideal transformer, it is assumed that there are no energy losses, and, hence, all the power at the primary winding is transferred to the secondary winding. However, in reality, the transformers always have some energy losses, and, hence, the output power obtained at the secondary winding is less than the input power at the primary winding due to energy losses.
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
820
The Ideal Transformer
344
In single-phase two-winding transformers, two windings are coiled around a magnetic core characterized by cross-sectional area A and magnetic permeability μ. A phasor current i1 enters the left winding while i2 exits the right winding, establishing the fundamental working of the transformer through electromagnetic principles.
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
344
Transformers in Distribution System
98
Transformers in distribution systems can be broadly categorized into distribution substation transformers and other distribution transformers. They are crucial for stepping down high transmission voltages to levels suitable for distribution and end-user applications.
Distribution substation transformers come in various ratings and typically use mineral oil for insulation and cooling. To prevent moisture and air from entering the oil, some transformers use an inert gas like nitrogen to fill the...
Distribution substation transformers come in various ratings and typically use mineral oil for insulation and cooling. To prevent moisture and air from entering the oil, some transformers use an inert gas like nitrogen to fill the...
98
Transformers with Off-Nominal Turns Ratios
132
In scenarios involving parallel transformers with disparate ratings, developing per-unit models requires accommodating off-nominal turns ratios. This situation arises when the selected base voltages are not proportional to the transformer’s voltage ratings. Consider a transformer where the rated voltages are related by the term a. If the chosen voltage bases satisfy a relationship involving term b, term c is defined as the ratio of these bases. This ratio is then substituted into the...
132
