Understanding Convolutional Neural Networks With Information Theory: An Initial Exploration

Area of Science:

• Information theory
• Machine learning
• Deep learning

Background:

• A novel functional estimator for Rényi's α-entropy was recently proposed.
• Its utility and applications are not widely known to practitioners.

Purpose of the Study:

• To demonstrate the straightforward measurement of information flow in convolutional neural networks (CNNs) using the novel estimator.
• To introduce the partial information decomposition (PID) framework for analyzing convolutional layer representations.
• To develop quantities for analyzing synergy and redundancy in CNNs.

Main Methods:

• Utilized a novel functional estimator based on the normalized eigenspectrum of a Hermitian matrix in a reproducing kernel Hilbert space (RKHS).
• Applied the estimator to measure information flow in CNNs without approximation.
• Integrated the partial information decomposition (PID) framework.

Main Results:

• Enabled straightforward, non-approximate measurement of information flow in realistic CNNs.
• Developed three quantities to analyze synergy and redundancy in convolutional layer representations.
• Validated two fundamental data processing inequalities.
• Revealed inner properties of CNN training.

Conclusions:

• The novel estimator provides a practical tool for analyzing information flow in CNNs.
• The PID framework offers new insights into the internal workings of convolutional layers.
• The findings contribute to understanding information processing within deep learning models.