Conditional Mutual Information Constrained Deep Learning for Classification

Area of Science:

• Machine Learning
• Deep Learning
• Information Theory

Background:

• Classification deep neural networks (DNNs) performance is evaluated using output probability distributions.
• Existing methods lack robust metrics for measuring intraclass concentration and interclass separation.

Purpose of the Study:

• Introduce conditional mutual information (CMI) and normalized CMI (NCMI) to quantify DNN concentration and separation.
• Propose a modified deep learning framework (CMIC-DL) to optimize these metrics.

Main Methods:

• Defined CMI for intraclass concentration and NCMI for interclass separation.
• Developed a CMI constrained deep learning (CMIC-DL) framework with an alternating learning algorithm.
• Evaluated popular DNNs on CIFAR-100 and ImageNet datasets.

Main Results:

• Validation accuracy of DNNs is inversely proportional to NCMI values.
• CMIC-DL trained DNNs outperform standard deep learning models in accuracy.
• CMIC-DL enhances DNN robustness against adversarial attacks.

Conclusions:

• CMI and NCMI offer effective measures for DNN classification performance.
• The CMIC-DL framework improves both accuracy and adversarial robustness.
• Visualizing learning via CMI/NCMI aids in understanding DNN training dynamics.