Uncovering Neural Learning Dynamics Through Latent Mutual Information.

Area of Science:

• Computer Science
• Machine Learning
• Artificial Intelligence

Background:

• Convolutional Neural Networks (CNNs) are key to image classification.
• Understanding information flow within CNNs during learning is crucial.

Purpose of the Study:

• To investigate how CNNs reorganize information during natural image classification.
• To analyze the role of mutual information (MI) in CNN representations.

Main Methods:

• Tracking mutual information between inputs, intermediate representations, and labels in VGG-16, ResNet-18, and ResNet-50.
• Analyzing information concentration in specific channels using knockouts, shuffles, and perturbations.
• Implementing a dependence-aware regularizer based on the Hilbert-Schmidt Independence Criterion.

Main Results:

• Label-relevant MI increases with network depth, while input MI varies with architecture.
• Information concentrates in a subset of channels, which are functionally necessary for accuracy.
• A regularizer promoting selective concentration improves convergence and accuracy.

Conclusions:

• CNN representation learning is driven by selective concentration and decorrelation, not global compression.
• Targeted regularization can guide CNNs towards efficient information processing.
• Findings offer insights into designing more effective deep learning models.