PWSNAS: Powering Weight Sharing NAS With General Search Space Shrinking Framework

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Neural Architecture Search (NAS) relies on efficient performance estimators.
• Weight-sharing methods (e.g., DARTS, One-Shot) speed up NAS by reusing parameters but suffer from inaccurate performance estimation.
• Existing NAS methods face challenges with unstable performance estimation.

Purpose of the Study:

• To propose PWSNAS, a general framework for improving weight-sharing NAS.
• To enhance the accuracy and stability of performance estimation in NAS.
• To reduce the difficulty of finding superior architectures by simplifying the search space.

Main Methods:

• PWSNAS automatically shrinks the search space by discarding less important candidate operators.
• Introduces two novel strategies: an angle-based metric to detect redundant operators and adjusting weight-sharing degrees.
• Progressively simplifies the original search space to create a smaller, more promising search space.

Main Results:

• Experiments on NASBench-201 validate the superiority of the proposed angle-based metric over accuracy-based and magnitude-based metrics.
• PWSNAS demonstrates consistent performance gains when applied to state-of-the-art NAS methods like SPOS, FairNAS, ProxylessNAS, DARTS, and PDARTS.
• The framework effectively reduces the search space, aiding in the discovery of superior architectures.

Conclusions:

• PWSNAS offers a general and effective framework for powering weight-sharing NAS.
• The proposed shrinking strategies improve the efficiency and accuracy of NAS.
• PWSNAS is compatible with various existing NAS methods, leading to performance enhancements.