Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Neural Architecture Search (NAS) is computationally expensive due to training requirements.
• Zero-shot NAS methods aim to predict network performance without parameter training.
• Existing zero-shot proxies are inspired by deep learning theory and show promise.

Purpose of the Study:

• To comprehensively review and compare state-of-the-art zero-shot NAS approaches.
• To emphasize the hardware awareness of these zero-shot NAS methods.
• To identify potential improvements for future proxy designs.

Main Methods:

• Review of mainstream zero-shot NAS proxies and their theoretical foundations.
• Large-scale experimental comparison of zero-shot proxies.
• Evaluation in both hardware-aware and hardware-oblivious NAS settings.

Main Results:

• Zero-shot NAS proxies can effectively predict network accuracy without training.
• Demonstrated effectiveness of reviewed proxies in diverse NAS scenarios.
• Highlighted the importance and impact of hardware awareness in zero-shot NAS.

Conclusions:

• Zero-shot NAS offers a viable, efficient alternative to traditional NAS.
• Further research into designing improved zero-shot proxies is warranted.
• Hardware-aware proxy design is crucial for practical NAS applications.