关于深度学习动态的两阶段视角
Robert de Mello Koch1,2, Animik Ghosh1
1Huzhou University, School of Science, Huzhou 313000, China.
Physical review. E
|September 16, 2025
概括
深度神经网络学习涉及快速的曲线拟合,然后缓慢的压缩. 这两阶段的过程,观察到grokking和双下降,对于概括至关重要.
科学领域:
- 机器学习 机器学习
- 深度学习理论 深度学习理论
- 计算神经科学是一种神经科学.
背景情况:
- 深度神经网络 (DNN) 呈现出复杂的学习动态.
- 像,双下降和信息瓶这样的现象表明非微不足道的零训练后错误行为.
- 了解DNN学习的时间结构是改善概括的关键.
研究的目的:
- 提出和验证深度神经网络学习的两阶段模型:曲线拟合和压缩.
- 为了证明 grokking,双降落和信息瓶的共享时间结构.
- 确定DNN学习的有效进度指标,并探索压缩阶段的作用.
主要方法:
- 在不同环境中对DNN培训动态的实证分析.
- 测量隐藏层和输入数据之间的相互信息.
- 信息理论指标与基于电路的指标 (局部复杂性,线性映射数) 的比较.
主要成果:
- DNN学习始终遵循两阶段的轨迹:快速的初始拟合,然后慢慢的压缩.
- 格罗克,双重降落和信息瓶现象的时间尺度在经验上一致.
- 相互信息是学习进度的强有力的衡量标准,与概括相关.
结论:
- 深度学习通过不同的曲线拟合和压缩阶段进行.
- 压缩阶段,类似于重新规范化组粗粒度,对于概括至关重要.
- 标准训练可能无法优化这种压缩阶段,这表明了算法改进的潜力.
相关概念视频
Introduction to Learning
961
Learning is the process of acquiring knowledge or skills through practice or experience, leading to long-lasting behavioral changes. This acquisition occurs through interaction with the environment and requires practice or experience. For instance, mastering a skill such as surfing requires considerable practice and experience, highlighting the essential role of repeated interactions with the environment in learning.
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
961
Parameters Affecting Nonlinear Elimination: Zero-Order Input, First-Order Absorption and Two-Compartment Model
292
Drugs administered through various routes can lead to nonlinear elimination, resulting in complex pharmacokinetic behaviors crucial to understanding efficacious drug dosing.
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
292
Observational Learning
841
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
841
Phase Transitions
22.5K
Whether solid, liquid, or gas, a substance's state depends on the order and arrangement of its particles (atoms, molecules, or ions). Particles in the solid pack closely together, generally in a pattern. The particles vibrate about their fixed positions but do not move or squeeze past their neighbors. In liquids, although the particles are closely spaced, they are randomly arranged. The position of the particles are not fixed—that is, they are free to move past their neighbors to...
22.5K
Multi-input and Multi-variable systems
394
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence of...
In the absence of...
394
Phase-lead and Phase-lag Controllers
531
Understanding the working function of different types of controllers can be illustrated with practical analogies, such as adjusting a stereo's volume equalizer. Cranking up the bass involves a phase-lead controller, which functions as a high-pass filter, while increasing the treble uses a phase-lag controller, which acts as a low-pass filter. PD controllers, similar to high-pass filters, enhance the system's response to high-frequency components. PI controllers, akin to low-pass...
531

