A Gradient-Guided Evolutionary Neural Architecture Search

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Neural Architecture Search (NAS) automates deep neural network design but is computationally intensive.
• Existing NAS methods face limitations in efficiency and performance optimization.

Purpose of the Study:

• To propose a novel hybrid algorithm, Gradient-guided Evolutionary NAS (GENAS), for efficient Convolutional Neural Network (CNN) design.
• To address the computational expense and limitations of traditional NAS and gradient descent methods.

Main Methods:

• GENAS combines evolutionary global and local search operators on a population of subnets sampled from a supernet.
• Candidate architectures are encoded in a table, manipulated using novel crossover and mutation operators.
• A local search inspired by differentiable NAS is applied to candidate architectures without retraining.

Main Results:

• GENAS achieved low test errors: 2.45% (CIFAR-10), 16.86% (CIFAR-100), and 23.9% (ImageNet).
• The method demonstrated significant computational efficiency, requiring only 0.26 GPU days.
• Decoupled subnet evaluation prevented strong coupling of operations within the supernet.

Conclusions:

• GENAS effectively expedites the training and evaluation processes in NAS.
• The proposed method successfully obtains high-performance CNN structures for image classification.
• GENAS overcomes limitations of purely evolutionary or gradient descent NAS approaches.