Autorep: Automatic network search with structured reparameterized based linear operation expansion and gradient proxy...

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) excel in computer vision but have high computational demands.
• Designing efficient, lightweight neural network architectures remains a significant challenge.
• Existing methods for automated neural architecture search (NAS) often struggle with efficiency and performance trade-offs.

Purpose of the Study:

• To improve the performance of one-shot neural architecture search (NAS) algorithms.
• To introduce a specific structural reparameterization strategy during SuperNet training.
• To develop methods for creating efficient, lightweight neural network architectures.

Main Methods:

• Employed structural reparameterization within SuperNet training to expand candidate operations into equivalent branches.
• Implemented an operation reduction strategy to remove low-effect extended linear layers.
• Utilized a prior sampling strategy and SynFlow proxy for efficient subnetwork validation and selection.

Main Results:

• The proposed strategy effectively utilizes representation potential during SuperNet training.
• Operation reduction and prior sampling strategies alleviate training difficulties and control computational costs.
• The method facilitates the design of lightweight architectures without significant performance degradation.

Conclusions:

• Structural reparameterization is a viable strategy to enhance one-shot NAS.
• The developed techniques balance model efficiency and performance for computer vision tasks.
• This approach offers a promising direction for creating practical and computationally feasible deep learning models.