Rethinking cell-based neural architecture search: A theoretical perspective.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cell-based neural architecture search (NAS) is a key area in deep learning.
• Understanding the theoretical underpinnings of NAS search spaces is crucial for developing efficient algorithms.
• Existing NAS methods often rely on empirical evaluations without deep theoretical guarantees.

Purpose of the Study:

• To theoretically analyze the importance of different architectures within a NAS search space.
• To determine if an optimal cell identified in isolation remains optimal when stacked.
• To provide theoretical justification for the cell-based NAS paradigm.

Main Methods:

• Modeling cell architectures as directed acyclic graphs (DAGs) with stem paths and bypass edges.
• Proving theoretical properties regarding the impact of adding skip connections and learnable operations to bypass edges.
• Analyzing the optimality of cell stacks based on greedy bypass edge addition.
• Empirically verifying theoretical findings using the NAS-BENCH-201 dataset.

Main Results:

• Adding skip connections or learnable operations to bypass edges in single cells generally decreases or maintains training loss.
• Certain architectures with specific weights show learnable operations performing worse than skip connections.
• A greedily constructed optimal cell structure guarantees an optimal cell stack when cells are identical.
• Network size and non-tied cell structures can impact the optimality of stacked cells.

Conclusions:

• Theoretical analysis supports the cell-based NAS paradigm by demonstrating the optimality of greedily constructed cells in stacks.
• The optimality of a cell is context-dependent, especially when considering factors beyond minimal training loss.
• Further research into flexible cell stacking strategies is warranted for improved performance.