Exploring Complicated Search Spaces With Interleaving-Free Sampling

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Conventional neural architecture search (NAS) algorithms are constrained by search spaces featuring only short-distance node connections.
• These limitations hinder the exploration of more effective and complex network architectures.
• Existing weight-sharing search algorithms fail in complicated search spaces due to interleaved connections (ICs).

Purpose of the Study:

• To investigate the efficacy of search algorithms within complex search spaces incorporating long-distance connections.
• To address the failure of existing weight-sharing algorithms caused by interleaved connections (ICs).
• To introduce a novel algorithm, Interleaving-Free Neural Architecture Search (IF-NAS), designed for enhanced architecture exploration.

Main Methods:

• Exploration of a complicated search space with long-distance connections.
• Development of the Interleaving-Free Neural Architecture Search (IF-NAS) algorithm.
• Implementation of a periodic sampling strategy to construct subnetworks, preventing ICs.

Main Results:

• IF-NAS significantly outperforms random sampling and previous weight-sharing algorithms in the proposed search space.
• The algorithm demonstrates effective generalization to microcell-based search spaces.
• The study highlights the critical role of macrostructure in neural architecture search.

Conclusions:

• IF-NAS offers a robust solution for exploring complex neural network architectures with long-distance connections.
• The periodic sampling strategy effectively mitigates issues caused by interleaved connections.
• This research underscores the importance of macrostructural considerations in advancing neural architecture search.