Good Performance Estimation Strategies are All You Need in Neural Architecture Search.

Area of Science:

• Artificial Intelligence
• Computer Science

Background:

• Neural Architecture Search (NAS) relies on Performance Estimation (PE) to evaluate architectures.
• Kendall's τ is the standard metric for evaluating PE strategies, but its optimality is questioned.

Purpose of the Study:

• To critically analyze Kendall's τ for PE in NAS.
• To propose a novel criterion, Minimum Keeping Ratio (MKR), for evaluating PE strategies.
• To develop an improved NAS method based on the new criterion.

Main Methods:

• Theoretical analysis and extensive experiments were conducted to evaluate Kendall's τ.
• The Minimum Keeping Ratio (MKR) criterion was developed and applied.
• A simple NAS method was derived by integrating PE strategies with random sampling.

Main Results:

• Kendall's τ was found to be suboptimal for evaluating PE strategies.
• The MKR criterion provides a unified perspective for comparing PE approaches.
• The proposed simple random sampling NAS method achieved state-of-the-art results on multiple benchmarks (NASbenchMacro, NASbench201, NASbench301).
• The method generalized to different search spaces (MobileNet) and tasks (semantic segmentation), achieving superior performance on ADE20K.

Conclusions:

• The Minimum Keeping Ratio (MKR) is a more principled and effective criterion for evaluating PE strategies in NAS.
• A simple random sampling NAS method integrating PE strategies demonstrates high efficiency and effectiveness.
• The proposed approach achieves state-of-the-art results and generalizes well across various NAS tasks and search spaces.