On Neural Networks Fitting, Compression, and Generalization Behavior via Information-Bottleneck-like Approaches

Area of Science:

• Artificial Intelligence
• Machine Learning
• Information Theory

Background:

• Neural network learning dynamics, including fitting, compression, and generalization, remain poorly understood.
• Estimating mutual information in high-dimensional spaces for neural networks is challenging.

Purpose of the Study:

• To propose a novel approach for capturing neural network learning dynamics.
• To replace difficult-to-estimate mutual information measures with more tractable ones.

Main Methods:

• Utilized information-bottleneck-type techniques.
• Replaced mutual information with minimum mean-squared error for input reconstruction and cross-entropy for classification.
• Conducted empirical studies across various network models, learning algorithms, and datasets.

Main Results:

• The proposed method reliably captures neural network dynamics during training and testing, outperforming classical information bottleneck approaches.
• Fitting and compression phases were observed consistently across different activation functions.
• Better generalization correlated with more pronounced fitting and compression phases.

Conclusions:

• The novel approach offers a more reliable way to analyze neural network learning dynamics.
• Fitting and compression are fundamental phases in neural network learning, influenced by architecture, training, and data.
• Understanding these phases is key to improving model generalization.