One-Shot Neural Architecture Search by Dynamically Pruning Supernet in Hierarchical Order

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Neural Architecture Search (NAS) automates neural network design but often requires extensive training.
• One-shot NAS methods train a single supernet for efficiency but face challenges with predictive evaluation.
• Pruning supernets during search is a promising technique, yet optimal pruning directions in complex spaces are underexplored.

Purpose of the Study:

• To investigate the role of path dropout in supernet training for NAS.
• To develop a novel pruning strategy for enhancing the efficiency and predictability of one-shot NAS.
• To propose the Hierarchically-Ordered Pruning Neural Architecture Search (HOPNAS) algorithm.

Main Methods:

• Revisiting path dropout strategy for supernet training, focusing on dropping neural operations.
• Observing and analyzing the characteristics of supernets trained with dropout.
• Developing the HOPNAS algorithm for dynamic supernet pruning with a defined pruning direction.

Main Results:

• Identified interesting characteristics of supernets trained with path dropout.
• Demonstrated that HOPNAS dynamically prunes the supernet effectively.
• Achieved competitive performance against state-of-the-art methods on CIFAR10 and ImageNet benchmarks.

Conclusions:

• Path dropout offers valuable insights into supernet training for NAS.
• HOPNAS provides an effective and efficient approach to neural architecture search.
• The proposed method shows strong performance and potential for complex search spaces.