OStr-DARTS: Differentiable Neural Architecture Search Based on Operation Strength

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Differentiable architecture search (DARTS) is an efficient method for neural architecture search (NAS).
• DARTS optimizes a supernet with mixed operations and selects the best architecture based on operation contribution.
• A key challenge in DARTS is the degeneration issue, leading to suboptimal architectures.

Purpose of the Study:

• To investigate the cause of the DARTS degeneration issue.
• To propose a novel operation selection criterion to mitigate the degeneration issue.
• To demonstrate the effectiveness of the proposed method in improving DARTS stability.

Main Methods:

• Proposed a new selection criterion based on operation strength, estimating importance by impact on final loss.
• Replaced the conventional magnitude-based selection method with the proposed criterion.
• Validated the method on NAS-Bench-201 and DARTS search spaces without modifying supernet optimization.

Main Results:

• The proposed operation strength criterion effectively addresses the DARTS degeneration issue.
• The method improves the stability of DARTS without altering the supernet optimization process.
• Experimental results on NAS-Bench-201 and DARTS search spaces confirm the method's effectiveness.

Conclusions:

• The magnitude-based selection method is a critical factor contributing to DARTS instability.
• The proposed operation strength criterion offers a viable solution to the DARTS degeneration problem.
• This research provides a more robust approach to neural architecture search using DARTS.