A selection perturbative hyper-heuristic for neural architecture search

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Neural Architecture Search (NAS) traditionally explores design spaces directly.
• Existing methods like genetic algorithms face limitations in exploring the full architecture spectrum.
• A need exists for alternative search strategies to overcome these limitations.

Purpose of the Study:

• To investigate an alternative approach to NAS by exploring the heuristic space.
• To introduce a hyper-heuristic method for indirectly searching the design space.
• To evaluate the efficacy of this indirect search strategy.

Main Methods:

• Introduction of the NAS operator space (NOS).
• Development of a Single Point Selection Perturbative Hyper-heuristic for NAS (SPHH-NAS).
• Utilization of a choice function for heuristic selection and Adaptive Improvement Limited Target Acceptance (AILTA) for move acceptance.

Main Results:

• SPHH-NAS successfully explores the heuristic space, indirectly mapping to the NOS and then the design space.
• The approach demonstrated superior performance compared to the majority of previous methods on NAS-101, NAS-201, and NAS-301 benchmark sets.
• Evaluation on two real-world datasets confirmed the effectiveness of SPHH-NAS.
• A significant reduction in computational cost was observed with SPHH-NAS.

Conclusions:

• Indirectly searching the design space via a heuristic space can access unexplored areas.
• SPHH-NAS offers a more efficient and effective alternative to direct search methods in NAS.
• The proposed method advances the field of automated machine learning by improving performance and reducing computational demands.