Neural architecture search applying optimal stopping theory

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Neural Architecture Search (NAS) demands significant computational resources, limiting accessibility for many researchers.
• Early-stopping random search methods show competitive performance in NAS.
• The impracticality of exploring large NAS search spaces necessitates efficient search strategies.

Purpose of the Study:

• To determine the optimal exploration percentage of NAS search spaces for efficient architecture discovery.
• To apply optimal stopping theory, specifically the Secretary Problem (SP), to NAS.
• To investigate variants of the SP for further reducing NAS search space exploration.

Main Methods:

• Applied the Secretary Problem (SP) and its extensions ('good enough', 'call back') to NAS.
• Trained and evaluated 672 unique architectures across 20,000 runs on MNIST and CIFAR-10 datasets.
• Statistically validated findings on NAS populations ranging from 100 to 3,500 architectures.

Main Results:

• Empirically and theoretically confirmed that exploring ~37% of the NAS search space is sufficient for acceptable architecture discovery.
• The 'good enough' and 'call back' SP variants reduced exploration to ~15% and ~4%, respectively.
• Results were statistically robust across various population sizes and numerous runs.

Conclusions:

• Researchers can balance computational costs and NAS effectiveness by exploring approximately 37% of the search space using SP principles.
• SP variants offer further reductions in computational requirements for NAS.
• Provides practical guidance for implementing efficient NAS strategies.