Allies Teach Better Than Enemies: Inverse Adversaries for Robust Knowledge Distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Adversarial robustness knowledge distillation aims to create smaller, robust models from larger ones.
• Existing methods struggle with transferring robustness due to potentially incorrect teacher predictions.
• This can negatively impact the student model's adversarial resilience.

Purpose of the Study:

• To develop a novel knowledge distillation scheme for improved adversarial robustness transfer.
• To address the limitations of previous methods by avoiding misguidance from incorrect teacher predictions.
• To enhance both natural and adversarial performance in compressed student models.

Main Methods:

• Introduced "inverse adversarial examples" by reversing adversarial perturbations to refine inputs.
• Developed a gradient matching mechanism using inverse adversaries for robust knowledge alignment.
• Proposed a weight-space disruption strategy to enhance robustness transfer between models.
• Investigated theoretical properties of inverse adversaries for insights into robustness transfer.

Main Results:

• Achieved state-of-the-art performance in both robustness and natural accuracy across various datasets.
• Outperformed prior methods by 3.8% on ImageNet for both clean and robust accuracy.
• Demonstrated that incorporating auxiliary data further boosts model robustness.
• Showcased the generalizability of the method to multimodal architectures.

Conclusions:

• The proposed adversarially robust knowledge distillation scheme effectively transfers robustness using inverse adversarial examples.
• The method provides theoretical insights into the connection between robustness and input gradients.
• This approach offers a significant advancement in creating efficient and robust AI models.