Glassy dynamics near the interpolation transition in deep recurrent networks

Area of Science:

• Machine Learning
• Deep Learning
• Statistical Physics

Background:

• Deep recurrent networks are complex models with behaviors influenced by network architecture.
• The interpolation transition marks a boundary between under- and over-parameterized regimes in deep learning.
• Understanding learning dynamics is crucial for optimizing model training and performance.

Purpose of the Study:

• To investigate the learning dynamics of deep recurrent networks near the interpolation transition.
• To analyze critical slowing down and aging phenomena during the training process.
• To draw parallels between deep learning dynamics and spin glass models.

Main Methods:

• Training deep recurrent networks on Bach chorales using stochastic gradient descent.
• Analyzing learning times and weight fluctuations near the depth-width boundary.
• Comparing observed phenomena with predictions from spin glass models.

Main Results:

• Observed critical slowing down of learning as network width approaches the critical value from the over-parameterized side.
• Identified the zero-loss limit of the critical width with the interpolation transition.
• Characterized aging in weight fluctuations, showing a scaling behavior in the under-parameterized phase that breaks down near the loss limit.

Conclusions:

• Deep recurrent network learning exhibits critical slowing down and aging, analogous to spin glass models.
• The interpolation transition is a key feature governing these learning dynamics.
• Spin glass models capture essential aspects of deep learning dynamics, offering insights for future research.