Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Information Theory

Background:

• Deep neural networks (DNNs) sometimes achieve superior performance via knowledge distillation compared to training from scratch.
• The underlying reasons for knowledge distillation's success remain an active area of research.

Purpose of the Study:

• To provide an information-theoretic perspective on knowledge distillation by quantifying knowledge points in DNN intermediate layers.
• To propose and verify hypotheses explaining why knowledge distillation improves DNN performance.

Main Methods:

• Quantifying knowledge points as input units with less discarded information in DNN layers.
• Developing metrics to analyze feature representations: quantity/quality of knowledge points, learning speed, and optimization stability.
• Testing hypotheses across diverse classification tasks (image, 3D point cloud, sentiment, question answering).

Main Results:

• DNNs trained with knowledge distillation encode more knowledge points than those trained from scratch.
• Knowledge distillation facilitates simultaneous learning of diverse knowledge points, unlike sequential learning from scratch.
• Optimization processes in knowledge distillation are generally more stable.

Conclusions:

• Knowledge distillation enhances DNNs by improving the quantity, quality, and learning dynamics of encoded knowledge points.
• The information-theoretic framework successfully explains the performance gains observed in knowledge distillation across various tasks.