Mitigating Accuracy-Robustness Trade-Off via Balanced Multi-Teacher Adversarial Distillation

Area of Science:

• Deep Learning
• Machine Learning Security
• Artificial Intelligence

Background:

• Adversarial Training enhances deep neural network robustness against attacks but often degrades performance on clean data.
• Existing knowledge distillation methods in Adversarial Training show limited improvements in clean accuracy.
• A significant accuracy-robustness trade-off persists in current deep learning defense strategies.

Purpose of the Study:

• To introduce a novel method, Balanced Multi-Teacher Adversarial Robustness Distillation (B-MTARD), to address the accuracy-robustness trade-off.
• To guide Adversarial Training using specialized teachers for clean and adversarial examples.
• To improve both the robustness and clean accuracy of deep neural networks.

Main Methods:

• B-MTARD employs a strong clean teacher and a strong robust teacher to process distinct data types.
• The Entropy-Based Balance algorithm is utilized to harmonize knowledge scales across teachers by maintaining consistent information entropy.
• The Normalization Loss Balance algorithm is proposed to regulate learning weights for balanced knowledge acquisition from multiple teachers.

Main Results:

• B-MTARD demonstrated superior performance compared to state-of-the-art methods on public datasets.
• The method effectively improved model robustness against various adversarial attacks.
• Experimental results validated the mitigation of the accuracy-robustness trade-off.

Conclusions:

• B-MTARD offers an effective solution for the accuracy-robustness dilemma in deep learning.
• The proposed balancing algorithms ensure consistent and effective knowledge transfer from multiple teachers.
• This approach advances the development of more robust and accurate deep neural networks.