Utilizing Information Bottleneck to Evaluate the Capability of Deep Neural Networks for Image Classification

Area of Science:

• Machine Learning
• Deep Learning
• Information Theory

Background:

• The Information Bottleneck (IB) principle offers insights into Deep Neural Network (DNN) analysis.
• Understanding the relationship between model accuracy and information-theoretic measures is crucial for DNN evaluation.

Purpose of the Study:

• To develop an information plane-based framework for evaluating DNN capabilities in image classification.
• To analyze the interplay between model accuracy, mutual information with input (I(X;T)), and mutual information with labels (I(T;Y)).

Main Methods:

• Focusing on the DNN's output (T) rather than hidden layers.
• Designing an information plane framework utilizing mutual information measures.
• Applying the framework to image classification tasks, including those with unbalanced data.

Main Results:

• The framework successfully evaluates DNNs (including CNNs) for image classification.
• Demonstrated effectiveness in model selection and determining sample requirements for unbalanced datasets.
• Provided explanations for the efficiency of transfer learning in deep learning.

Conclusions:

• The proposed information plane framework is effective for evaluating DNNs in image classification.
• The framework offers practical benefits for model selection and handling data imbalance.
• It enhances the understanding of transfer learning mechanisms in deep learning.