An Information Theoretic Interpretation to Deep Neural Networks.

Area of Science:

• Machine Learning
• Information Theory
• Computer Vision

Background:

• Deep neural networks (DNNs) achieve high performance by extracting informative features.
• The feature extraction process in DNNs lacks a formal theoretical understanding.
• Existing methods do not fully connect practical training with theoretical feature optimality.

Purpose of the Study:

• To formalize the intuition that DNNs extract informative features.
• To establish an information-theoretic framework for feature selection and evaluate DNN features.
• To develop a metric for assessing the effectiveness of extracted features.

Main Methods:

• Application of local information geometric analysis.
• Development of an information-theoretic framework for feature selection.
• Quantitative analysis of network structure's impact on feature extraction.
• Introduction of the H-score metric for feature evaluation.

Main Results:

• Demonstration of the information-theoretic optimality of DNN features.
• Characterization of how network structure influences feature extraction.
• Validation of the H-score's ability to link practical training with theory.
• Experimental validation on synthesized data and ImageNet.

Conclusions:

• The study provides a formal information-theoretic framework for understanding DNN feature extraction.
• The H-score offers a practical metric to evaluate feature effectiveness and guide network design.
• Theoretical insights and practical metrics are validated, advancing DNN research.