在深度神经网络中不断学习的顺序参数和阶段过渡
Haozhe Shan1,2,3, Qianyi Li1,4, Haim Sompolinsky1,5
1Center for Brain Science, Harvard University, Cambridge, MA 02138.
概括
人工神经网络中的持续学习 (CL) 面临着灾难性的遗忘. 本研究引入了统计力学理论,以基于任务相似性和网络架构来预测CL性能,为忘记缓解提供了洞察力.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 统计力学 统计力学
背景情况:
- 持续学习 (CL) 允许系统在不丢失先前信息的情况下获得新知识.
- 人工神经网络 (NN) 扎着灾难性的遗忘,在学习新任务后降低了以前任务的性能.
- 现有的CL缓解技术缺乏对故障机制的理论理解.
研究的目的:
- 开发一个统计力学理论的CL在深入,广泛的NNs.
- 描述任务关系和网络架构如何影响遗忘和干扰.
- 提供对CL性能和缓解策略的理论见解.
主要方法:
- 开发了一个统计力学理论来建模NN在顺序任务学习期间的输入输出映射.
- 引入了订单参数 (OPs) 来量化任务相似性和网络属性.
- 通过对基准任务的数值评估验证了理论预测.
主要成果:
- 对于单头CL,任务特征/规则相似性 (通过OP测量) 预测了CL行为;增加网络深度减少了干扰.
- 多头的CL表现出基于任务相似性的性能阶段过渡.
- 在多头CL中,低任务相似性可能会导致灾难性的前级干扰,损害概括性.
结论:
- 该理论成功地预测了跨不同架构和任务关系的CL性能.
- 网络深度和任务相似性是影响遗忘和干扰的关键因素.
- 这些发现为设计更有效的持续学习系统提供了理论指导.
相关概念视频
Phase Transitions
23.2K
Whether solid, liquid, or gas, a substance's state depends on the order and arrangement of its particles (atoms, molecules, or ions). Particles in the solid pack closely together, generally in a pattern. The particles vibrate about their fixed positions but do not move or squeeze past their neighbors. In liquids, although the particles are closely spaced, they are randomly arranged. The position of the particles are not fixed—that is, they are free to move past their neighbors to...
23.2K
Phase Transitions: Melting and Freezing
15.2K
Heating a crystalline solid increases the average energy of its atoms, molecules, or ions, and the solid gets hotter. At some point, the added energy becomes large enough to partially overcome the forces holding the molecules or ions of the solid in their fixed positions, and the solid begins the process of transitioning to the liquid state or melting. At this point, the temperature of the solid stops rising, despite the continual input of heat, and it remains constant until all of the solid is...
15.2K
Phase Transitions: Vaporization and Condensation
21.5K
The physical form of a substance changes on changing its temperature. For example, raising the temperature of a liquid causes the liquid to vaporize (convert into vapor). The process is called vaporization—a surface phenomenon. Vaporization occurs when the thermal motion of the molecules overcome the intermolecular forces, and the molecules (at the surface) escape into the gaseous state. When a liquid vaporizes in a closed container, gas molecules cannot escape. As these gas phase molecules...
21.5K
Phase Transitions: Sublimation and Deposition
20.2K
Some solids can transition directly into the gaseous state, bypassing the liquid state, via a process known as sublimation. At room temperature and standard pressure, a piece of dry ice (solid CO2) sublimes, appearing to gradually disappear without ever forming any liquid. Snow and ice sublimate at temperatures below the melting point of water, a slow process that may be accelerated by winds and the reduced atmospheric pressures at high altitudes. When solid iodine is warmed, the solid sublimes...
20.2K
Phase Diagrams
50.3K
A phase diagram combines plots of pressure versus temperature for the liquid-gas, solid-liquid, and solid-gas phase-transition equilibria of a substance. These diagrams indicate the physical states that exist under specific conditions of pressure and temperature and also provide the pressure dependence of the phase-transition temperatures (melting points, sublimation points, boiling points). Regions or areas labeled solid, liquid, and gas represent single phases, while lines or curves represent...
50.3K
Properties of Transition Metals
30.0K
Transition metals are defined as those elements that have partially filled d orbitals. As shown in Figure 1, the d-block elements in groups 3–12 are transition elements. The f-block elements, also called inner transition metals (the lanthanides and actinides), also meet this criterion because the d orbital is partially occupied before the f orbitals.
30.0K


