DNA Family: Boosting Weight-Sharing NAS With Block-Wise Supervisions

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Neural Architecture Search (NAS) automates the design of neural networks, a crucial aspect of advancing machine learning.
• Weight-sharing NAS methods offer efficiency but struggle with low search effectiveness due to unreliable architecture ratings within vast search spaces.

Purpose of the Study:

• To overcome the limitations of weight-sharing NAS by developing a more effective and scalable architecture rating mechanism.
• To introduce Distilling Neural Architecture (DNA) techniques for modularizing search spaces and improving NAS performance.

Main Methods:

• Modularized large search spaces into smaller, manageable blocks.
• Developed a family of models using Distilling Neural Architecture (DNA) techniques.
• Utilized generalization boundedness tools to analyze architecture rating trustworthiness.

Main Results:

• Achieved state-of-the-art top-1 accuracy of 78.9% on ImageNet for mobile convolutional networks and 83.6% for small vision transformers.
• Demonstrated the ability to rate all architecture candidates, unlike previous sub-space heuristic methods.
• Successfully sought architectures of varying depths and widths under computational constraints.

Conclusions:

• The proposed DNA family effectively resolves scalability, efficiency, and multi-modal compatibility issues in weight-sharing NAS.
• DNA techniques provide a robust solution for trustworthy architecture rating, significantly boosting NAS effectiveness.
• The findings offer valuable insights into improving neural architecture rating and NAS optimization.