Two-phase perspective on deep learning dynamics

Area of Science:

• Machine Learning
• Deep Learning Theory
• Computational Neuroscience

Background:

• Deep neural networks (DNNs) exhibit complex learning dynamics.
• Phenomena like grokking, double descent, and information bottleneck suggest non-trivial post-zero-training-error behavior.
• Understanding the temporal structure of DNN learning is key to improving generalization.

Purpose of the Study:

• To propose and validate a two-phase model of deep neural network learning: curve-fitting and compression.
• To demonstrate the shared temporal structure of grokking, double descent, and information bottleneck.
• To identify effective progress measures for DNN learning and explore the role of the compression phase.

Main Methods:

• Empirical analysis of DNN training dynamics in diverse settings.
• Measurement of mutual information between hidden layers and input data.
• Comparison of information-theoretic measures with circuit-based metrics (local complexity, linear mapping number).

Main Results:

• DNN learning consistently follows a two-phase trajectory: rapid initial fitting followed by slower compression.
• The timescales of grokking, double descent, and information bottleneck phenomena align empirically.
• Mutual information serves as a robust measure of learning progress, correlating with generalization.

Conclusions:

• Deep learning progresses through distinct curve-fitting and compression phases.
• The compression phase, analogous to renormalization group coarse-graining, is critical for generalization.
• Standard training may not optimize this compression phase, suggesting potential for algorithmic improvements.