$\beta $-DARTS++: Bi-Level Regularization for Proxy-Robust Differentiable Architecture Search.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Neural Architecture Search (NAS) automates neural network design, with differentiable NAS (e.g., DARTS) offering efficiency.
• Existing differentiable NAS methods face challenges in stability, generalization, and robustness to proxy configurations.

Purpose of the Study:

• To introduce Beta-Decay regularization to stabilize DARTS-based NAS and improve the generalization of searched architectures.
• To address the robustness of differentiable NAS methods across diverse proxy settings.
• To propose Bi-level regularization by combining Beta-Decay with flooding regularization for enhanced proxy robustness.

Main Methods:

• Introduced Beta-Decay regularization to constrain architecture parameter values and variance during DARTS training.
• Conducted comprehensive experiments to benchmark differentiable NAS methods across various proxy configurations (data, channels, layers, epochs).
• Integrated flooding regularization into the weight optimization of Beta-Decay DARTS, creating Bi-level regularization.

Main Results:

• Beta-Decay regularization significantly stabilizes the NAS search process and improves the transferability of searched networks.
• Beta-Decay DARTS (β-DARTS) demonstrated superior performance across most proxy settings compared to other NAS methods.
• Bi-level regularization further enhanced the proxy robustness of differentiable NAS methods, validated both experimentally and theoretically.

Conclusions:

• Beta-Decay is an effective regularization technique for improving the stability and generalization of DARTS-based NAS.
• The proposed β-DARTS method exhibits strong robustness to various proxy configurations.
• Bi-level regularization offers a promising direction for enhancing the proxy robustness of differentiable NAS.