Neural Architecture Search via Proxy Validation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Current neural architecture search (NAS) methods identify optimal architectures based on validation examples and network weights.
• Intermediate validation results during NAS are valuable but underutilized.
• Existing NAS methods can be computationally intensive.

Purpose of the Study:

• To propose a new method for approximating the validation loss landscape to efficiently identify optimal neural architectures.
• To enhance the efficiency of neural architecture search.
• To leverage intermediate validation results more effectively.

Main Methods:

• Approximating the validation loss landscape by learning a mapping from neural architectures to validation losses.
• Developing a novel architecture sampling strategy for efficient approximation.
• Utilizing an operation importance weight (OIW) to balance sampling randomness and certainty.
• Learning neural architecture representations via a graph autoencoder (GAE).

Main Results:

• The proposed proxy validation loss landscape effectively identifies optimal neural architectures.
• The novel sampling strategy improves the efficiency of NAS.
• The method demonstrates effectiveness in both differentiable NAS and evolutionary-algorithm-based (EA-based) NAS.

Conclusions:

• Approximating the validation loss landscape is an effective strategy for efficient neural architecture search.
• The proposed methods offer significant improvements in NAS efficiency and performance.
• This approach provides a valuable tool for discovering optimal neural network designs.