Adversarial Distillation for Learning with Privileged Provisions.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Knowledge distillation trains compact student models from larger teacher models.
• Traditional methods struggle with student models learning the true data distribution.
• Resource constraints necessitate efficient model training techniques.

Purpose of the Study:

• To develop an improved knowledge distillation technique for resource-constrained environments.
• To enhance the ability of student models to learn the real data distribution.
• To accelerate the training convergence of knowledge distillation.

Main Methods:

• Proposed adversarial distillation with a student, teacher, and multi-class discriminator.
• Utilized adversarial and distillation losses for simultaneous optimization.
• Employed the Gumbel-Softmax trick for low-variance gradient updates from the discriminator.

Main Results:

• Adversarial distillation demonstrated superior performance compared to traditional methods.
• Significant improvements in both model accuracy and training speed were observed.
• The Gumbel-Softmax trick effectively accelerated discriminator training.

Conclusions:

• Adversarial distillation offers a more effective approach to training student models.
• The proposed method successfully addresses limitations of traditional knowledge distillation.
• This technique is highly beneficial for applications with computational constraints.