On neural architecture search and hyperparameter optimization: A max-flow based approach

Automated Machine Learning (AutoML) involves the automatic production of models for specific tasks on given datasets, which can be divided into two aspects: Neural Architecture Search (NAS) for model construction and Hyperparameter Optimization (HPO) for model training. One of the most important components in an AutoML strategy is the search algorithm, which aims to recommend effective configurations according to historical observations. In this work, we propose a novel max-flow based search algorithm for AutoML by representing NAS and HPO as a Max-Flow problem on a graph and thus derive a couple of novel AutoML strategies, dubbed MF-NAS and MF-HPO, which handle the search space and the search strategy graphically. To be specific, MF-NAS induces parallel edges with capacities by combining different operations such as skip connections, convolutions, and pooling, whereas MF-HPO allows parallel edges to be regarded as intervals within the combined search spaces. The learned weights and capacities of the parallel edges are alternately updated during the search process. To make MF-NAS and MF-HPO more efficient, we implement a semi-synchronous search mode for NAS and a warmup scheme for HPO, respectively. We conduct extensive experiments to evaluate the competitive efficacy and efficiency of our proposed MF-NAS and MF-HPO across different datasets and search spaces.