Neural Architecture Transfer

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Neural Architecture Search (NAS) automates neural network design but is computationally expensive for each deployment.
• Existing methods require separate searches for different hardware or objectives, limiting practical application.
• A need exists for efficient methods to generate task-specific models adaptable to multiple, potentially conflicting, objectives.

Purpose of the Study:

• To introduce Neural Architecture Transfer (NAT), a method for efficient, multi-objective neural network design.
• To enable the generation of specialized subnets from learned supernets without additional training.
• To significantly reduce the computational cost associated with neural architecture search.

Main Methods:

• Learning task-specific supernets from which specialized subnets can be sampled.
• Integrating online transfer learning with a many-objective evolutionary search procedure.
• Iteratively adapting a pre-trained supernet while simultaneously searching for task-specific subnets.

Main Results:

• NATNets demonstrated state-of-the-art performance on 11 benchmark image classification tasks, including ImageNet, under mobile constraints (≤ 600M Multiply-Adds).
• Small-scale, fine-grained datasets showed the most significant improvements.
• The NAT approach achieved orders of magnitude greater efficiency compared to existing NAS methods.

Conclusions:

• NAT offers a computationally efficient and effective alternative to conventional transfer learning for neural network design.
• The method successfully generates competitive, task-specific models for diverse image classification tasks and computational objectives.
• NAT represents a significant advancement in automating neural network design for practical, multi-objective applications.