Towards a configurable and non-hierarchical search space for NAS

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Neural Architecture Search (NAS) surpasses manual Neural Network (NN) design but often relies on rigid, predefined search spaces and hierarchical structures.
• Adapting existing NAS methods to new problems is challenging and hinders understanding the impact of the search algorithm versus the architecture itself.

Purpose of the Study:

• To develop a more flexible NAS methodology with a customizable search space for comprehensive network exploration.
• To reduce reliance on expert knowledge in designing Neural Network architectures.
• To enable complexity-aware NAS for optimizing both performance and model size.

Main Methods:

• Utilizes Gaussian Process (GP)-based Bayesian Optimization (BO) within a continuous architecture embedding space.
• Employs a Wasserstein Autoencoder for the embedding, regularized by Maximum Mean Discrepancy (MMD) penalization.
• Incorporates a Fully Input Convex Neural Network (FICNN) latent predictor to estimate architecture parameter counts.

Main Results:

• The embedding's effectiveness for optimization was validated on tasks like minimizing parameters and maximizing a zero-shot accuracy proxy.
• Two complexity-aware NAS variants were executed on CIFAR-10 and STL-10 datasets using distinct search spaces.
• The methodology successfully identified competitive Neural Network architectures with constrained model sizes.

Conclusions:

• The proposed NAS methodology offers enhanced flexibility and reduces the need for expert-defined search spaces.
• This approach facilitates efficient optimization of Neural Network architectures, balancing performance with model complexity.
• The findings demonstrate the potential for automated, adaptable Neural Network design in machine learning.