Deep Neural Network Self-Distillation Exploiting Data Representation Invariance.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Existing methods for creating accurate small networks rely on model compression or knowledge distillation, often requiring complex assistive models.
• These assistive models increase training complexity, memory usage, and time costs.

Purpose of the Study:

• To propose a self-distillation (SD) mechanism for directly obtaining high-accuracy models without an assistive model.
• To leverage the concept of data representation invariance inspired by human vision.

Main Methods:

• Developed a self-distillation (SD) learning algorithm where a single network learns from its own distorted versions.
• Utilized maximum mean discrepancy to ensure global feature consistency.
• Employed Kullback-Leibler divergence to maintain posterior class probability consistency across distorted branches.

Main Results:

• The proposed SD method effectively reduces generalization error across various network architectures (AlexNet, VGGNet, ResNet, etc.).
• Achieved superior performance compared to existing model distillation methods on benchmark datasets (MNIST, CIFAR-10/100, ImageNet).
• Demonstrated minimal extra training effort required for the SD approach.

Conclusions:

• Self-distillation offers an efficient and effective approach to developing high-accuracy small networks.
• The method successfully learns data representation invariance, leading to improved model generalization.
• This technique provides a viable alternative to traditional knowledge distillation, reducing computational overhead.