RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation

Area of Science:

• Artificial Intelligence
• Machine Learning

Background:

• Deep Neural Networks (DNNs) are susceptible to adversarial attacks.
• Neural Architecture Search (NAS) optimizes DNNs for prediction accuracy but its robustness against adversarial attacks is understudied, especially with knowledge distillation.
• Knowledge distillation typically focuses on matching outputs only at the final layer.

Purpose of the Study:

• To investigate if NAS can generate robust neural architectures by inheriting robustness from a strong teacher model.
• To propose a novel NAS algorithm, RNAS-CL, that leverages cross-layer knowledge distillation for improved adversarial robustness.

Main Methods:

• Developed Robust Neural Architecture Search by Cross-Layer knowledge distillation (RNAS-CL).
• RNAS-CL searches for optimal teacher layers to supervise corresponding student layers, unlike traditional methods focusing solely on the final output layer.
• Employed cross-layer knowledge distillation to transfer robustness from a pre-trained robust teacher network to a new architecture.

Main Results:

• RNAS-CL successfully produces neural architectures that are both compact and adversarially robust.
• The proposed cross-layer distillation approach effectively transfers robustness from the teacher model.
• Experimental results validate the effectiveness of RNAS-CL in enhancing model security.

Conclusions:

• RNAS-CL offers a new method for discovering neural architectures with enhanced adversarial robustness.
• The findings suggest that NAS can be guided to produce secure models through strategic knowledge distillation.
• This research opens avenues for developing more resilient deep learning systems for critical applications.