在PINN中学习:阶段过渡,扩散平衡和概括
Sokratis J Anagnostopoulos1, Juan Diego Toscano2, Nikolaos Stergiopulos1
1Laboratory of Hemodynamics and Cardiovascular Technology, EPFL, Lausanne, 1015, VD, Switzerland.
概括
我们在神经网络训练中发现了一个新的"扩散平衡"阶段, 这一阶段与同质残余相结合,可增强模型的概括性,加快学习速度.
科学领域:
- 深度学习的优化
- 神经网络动态
- 机器学习理论
背景情况:
- 了解神经网络的学习动态对于改善模型性能至关重要.
- 现有的理论描述了培训中的漂移和扩散阶段.
- 在这些阶段,渐变信号噪声比率 (SNR) 的作用需要进一步研究.
研究的目的:
- 使用梯度SNR研究完全连接的神经网络的学习动态.
- 识别和描述除了漂移和扩散之外的新培训阶段.
- 根据观察到的动态,提出改善概括和融合的方法.
主要方法:
- 对非凸的目标进行一级优化器分析.
- 通过信息瓶理论来解释培训阶段.
- 检查神经梯度的信号与噪声比 (SNR).
- 一个样本重量计方案的开发和测试
主要成果:
- 一个新品的识别
- 扩散平衡
- (DE) 阶段的特征是有序的神经梯度和稳定的收.
- DE阶段与增加的SNR和同质残留相对应,导致更好的概括.
- 一个拟议的样本重量计方案提高了残余均性和概括性.
- 在DE阶段过渡期间观察到和诱导的激活压缩.
- 物理信息神经网络 (PINNs) 的实验验证显示了更快的融合和泛化.
结论:
- 已确定的DE阶段为稳定的神经网络培训提供了新的视角.
- 在DE阶段沿着有序梯度实现均的残余值是优质概括的关键.
- 这些发现表明深度学习优化策略的潜在改进,特别是基于物理的方法.
相关概念视频
Phase Transitions
20.2K
Whether solid, liquid, or gas, a substance's state depends on the order and arrangement of its particles (atoms, molecules, or ions). Particles in the solid pack closely together, generally in a pattern. The particles vibrate about their fixed positions but do not move or squeeze past their neighbors. In liquids, although the particles are closely spaced, they are randomly arranged. The position of the particles are not fixed—that is, they are free to move past their neighbors to...
20.2K
Phase Transitions: Sublimation and Deposition
17.9K
Some solids can transition directly into the gaseous state, bypassing the liquid state, via a process known as sublimation. At room temperature and standard pressure, a piece of dry ice (solid CO2) sublimes, appearing to gradually disappear without ever forming any liquid. Snow and ice sublimate at temperatures below the melting point of water, a slow process that may be accelerated by winds and the reduced atmospheric pressures at high altitudes. When solid iodine is warmed, the solid sublimes...
17.9K
Phase Changes
4.5K
Phase transitions play an important theoretical and practical role in the study of heat flow. In melting or fusion, a solid turns into a liquid; the opposite process is freezing. In evaporation, a liquid turns into a gas; the opposite process is condensation.
A substance melts or freezes at a temperature called its melting point and boils or condenses at its boiling point. These temperatures depend on pressure. High pressure favors the denser form of the substance, so typically, high pressure...
A substance melts or freezes at a temperature called its melting point and boils or condenses at its boiling point. These temperatures depend on pressure. High pressure favors the denser form of the substance, so typically, high pressure...
4.5K
Dynamic Equilibrium
53.3K
A reversible chemical reaction represents a chemical process that proceeds in both forward (left to right) and reverse (right to left) directions. When the rates of the forward and reverse reactions are equal, the concentrations of the reactant and product species remain constant over time and the system is at equilibrium. A special double arrow is used to emphasize the reversible nature of the reaction. The relative concentrations of reactants and products in equilibrium systems vary greatly;...
53.3K
Generalization, Discrimination, and Extinction
784
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
784
Phase Diagram
6.1K
The phase of a given substance depends on the pressure and temperature. Thus, plots of pressure versus temperature showing the phase in each region provide considerable insights into the thermal properties of substances. Such plots are known as phase diagrams. For instance, in the phase diagram for water (Figure 1), the solid curve boundaries between the phases indicate phase transitions (i.e., temperatures and pressures at which the phases coexist).
6.1K


