Accelerating Zero-Shot NAS With Feature Map-Based Proxy and Operation Scoring Function.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Neural Architecture Search (NAS) automates model design but faces high computational costs and limitations in generating diverse architectures.
• Existing NAS methods often depend on gradients and data labels, leading to inefficiencies and discrepancies.
• The need for computationally efficient and diverse NAS methods is critical for advancing deep learning.

Purpose of the Study:

• To propose a novel zero-cost proxy, MeCo, for Neural Architecture Search (NAS).
• To develop an efficient zero-shot NAS scheme, FLASH, leveraging the MeCo proxy.
• To enhance the diversity and efficiency of architecture generation in NAS.

Main Methods:

• Introduced MeCo, a zero-cost proxy based on the Pearson correlation matrix of feature maps.
• Developed MeCo_opt, a variant of the MeCo proxy.
• Proposed FLASH, a zero-shot NAS scheme utilizing a proxy-based operation scoring function and a greedy heuristic.
• Evaluated MeCo and FLASH using a single forward pass with random data for efficient computation.

Main Results:

• MeCo and MeCo_opt require only one random data sample for computation, drastically reducing costs.
• FLASH constructs diverse model architectures, avoiding the repetition of cells common in other methods.
• FLASH demonstrated significant efficiency gains, being one to six orders of magnitude faster than state-of-the-art baselines.
• The proposed methods achieved the highest model accuracy compared to existing NAS techniques.

Conclusions:

• MeCo and FLASH offer a highly efficient and effective approach to Neural Architecture Search.
• The zero-cost proxy and zero-shot NAS scheme significantly advance the field by enabling diverse and accurate architecture generation.
• This work provides a computationally inexpensive yet powerful alternative for automated machine learning architecture design.