An architecture entropy regularizer for differentiable neural architecture search.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Differentiable architecture search (DARTS) is a popular neural architecture search (NAS) method enabling gradient-based optimization.
• However, DARTS suffers from poor stability and generalizability, hindering its practical application.
• This instability is linked to locally optimal architecture parameters arising from conflicting optimization objectives (Matthew effect and discretization discrepancy).

Purpose of the Study:

• To address the stability and generalizability issues in DARTS.
• To propose a novel regularization technique to mitigate the identified optimization dilemma.
• To enhance the performance of differentiable NAS algorithms.

Main Methods:

• Introduced 'architecture entropy' to quantify the discrepancy among architecture parameters of candidate operations.
• Utilized architecture entropy as a regularizer to control the learning of architecture parameters.
• Employed variable coefficients for the regularizer to balance conflicting optimization objectives.

Main Results:

• The proposed architecture entropy regularizer significantly improved the stability and generalizability of DARTS.
• Experiments demonstrated enhanced performance across various differentiable NAS algorithms, datasets, and search spaces.
• Achieved more accurate and robust results on benchmark datasets like CIFAR-10 and ImageNet.

Conclusions:

• Architecture entropy effectively resolves the dilemma causing instability in DARTS.
• The proposed regularizer offers a promising approach to improve differentiable NAS.
• This method leads to more reliable and high-performing neural network architectures.