You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Neural Architecture Search (NAS) is crucial for developing efficient deep learning models.
• Existing NAS methods face challenges due to large, non-continuous search spaces.
• Previous approaches often rely on evolutionary algorithms or reinforcement learning, which can be computationally intensive.

Purpose of the Study:

• To propose a novel and efficient Neural Architecture Search method called Direct Sparse Optimization NAS (DSO-NAS).
• To address the challenges of large and non-continuous search spaces in NAS.
• To leverage model pruning techniques for optimizing neural network architectures.

Main Methods:

• DSO-NAS reformulates the architecture search problem as a sparse optimization task.
• It introduces scaling factors to control information flow and applies sparse regularizations to prune connections.
• An efficient, theoretically sound optimization method is derived for solving the sparse optimization problem.

Main Results:

• DSO-NAS achieves 2.74% average test error on CIFAR-10.
• On ImageNet, it reaches 25.4% test error with 600M FLOPs, using 8 GPUs in 18 hours.
• The method demonstrates competitive performance on semantic segmentation tasks, such as on the PASCAL VOC dataset.

Conclusions:

• DSO-NAS offers an efficient and differentiable approach to neural architecture search.
• The method is applicable to large datasets and diverse tasks beyond classification.
• DSO-NAS provides a promising alternative to traditional NAS methods, achieving strong empirical results.