深層ニューラルネットワークにおける継続学習の順序パラメータと相転移
Haozhe Shan1,2,3, Qianyi Li1,4, Haim Sompolinsky1,5
1Center for Brain Science, Harvard University, Cambridge, MA 02138.
まとめ
人工ニューラルネットワーク(NN)における継続学習(CL)は、壊滅的な忘却に直面します。この研究は、タスクの類似性とネットワークアーキテクチャに基づいてCLパフォーマンスを予測するための統計力学理論を導入し、忘却緩和に関する洞察を提供します。
科学分野:
- 人工知能
- 機械学習
- 統計力学
背景:
- 継続学習(CL)により、システムは以前の情報を失うことなく新しい知識を獲得できます。
- 人工ニューラルネットワーク(NN)は、新しいタスクを学習した後に以前のタスクのパフォーマンスが低下する、壊滅的な忘却に苦しんでいます。
- 既存のCL緩和技術は、失敗メカニズムの理論的理解を欠いています。
主な方法:
- 逐次タスク学習中のNNの入出力マッピングをモデル化するための統計力学理論を開発しました。
- タスクの類似性とネットワークプロパティを定量化するために順序パラメータ(OP)を導入しました。
- ベンチマークタスクでの数値評価を通じて理論的予測を検証しました。
結論:
- この理論は、さまざまなアーキテクチャとタスクの関係にわたるCLパフォーマンスをうまく予測します。
- ネットワークの深さとタスクの類似性は、忘却と干渉に影響を与える重要な要因です。
- この発見は、より効果的な継続学習システムを設計するための理論的ガイダンスを提供します。
関連する概念動画
Phase Transitions
23.2K
Whether solid, liquid, or gas, a substance's state depends on the order and arrangement of its particles (atoms, molecules, or ions). Particles in the solid pack closely together, generally in a pattern. The particles vibrate about their fixed positions but do not move or squeeze past their neighbors. In liquids, although the particles are closely spaced, they are randomly arranged. The position of the particles are not fixed—that is, they are free to move past their neighbors to...
23.2K
Phase Transitions: Melting and Freezing
15.2K
Heating a crystalline solid increases the average energy of its atoms, molecules, or ions, and the solid gets hotter. At some point, the added energy becomes large enough to partially overcome the forces holding the molecules or ions of the solid in their fixed positions, and the solid begins the process of transitioning to the liquid state or melting. At this point, the temperature of the solid stops rising, despite the continual input of heat, and it remains constant until all of the solid is...
15.2K
Phase Transitions: Vaporization and Condensation
21.5K
The physical form of a substance changes on changing its temperature. For example, raising the temperature of a liquid causes the liquid to vaporize (convert into vapor). The process is called vaporization—a surface phenomenon. Vaporization occurs when the thermal motion of the molecules overcome the intermolecular forces, and the molecules (at the surface) escape into the gaseous state. When a liquid vaporizes in a closed container, gas molecules cannot escape. As these gas phase molecules...
21.5K
Phase Transitions: Sublimation and Deposition
20.2K
Some solids can transition directly into the gaseous state, bypassing the liquid state, via a process known as sublimation. At room temperature and standard pressure, a piece of dry ice (solid CO2) sublimes, appearing to gradually disappear without ever forming any liquid. Snow and ice sublimate at temperatures below the melting point of water, a slow process that may be accelerated by winds and the reduced atmospheric pressures at high altitudes. When solid iodine is warmed, the solid sublimes...
20.2K
Phase Diagrams
50.3K
A phase diagram combines plots of pressure versus temperature for the liquid-gas, solid-liquid, and solid-gas phase-transition equilibria of a substance. These diagrams indicate the physical states that exist under specific conditions of pressure and temperature and also provide the pressure dependence of the phase-transition temperatures (melting points, sublimation points, boiling points). Regions or areas labeled solid, liquid, and gas represent single phases, while lines or curves represent...
50.3K
Properties of Transition Metals
30.0K
Transition metals are defined as those elements that have partially filled d orbitals. As shown in Figure 1, the d-block elements in groups 3–12 are transition elements. The f-block elements, also called inner transition metals (the lanthanides and actinides), also meet this criterion because the d orbital is partially occupied before the f orbitals.
30.0K


