Surrogate-Assisted Hybrid-Model Estimation of Distribution Algorithm for Mixed-Variable Hyperparameters Optimization...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Convolutional Neural Network (CNN) performance is highly dependent on hyperparameter configuration.
• Optimizing CNN hyperparameters is challenging due to mixed-variable types, large search spaces, and high computational costs.

Purpose of the Study:

• To propose a novel Estimation of Distribution Algorithm (EDA) for efficient CNN hyperparameter optimization.
• To address the difficulties of mixed-variable hyperparameters, large search spaces, and expensive computational costs in hyperparameter tuning.

Main Methods:

• A hybrid-model EDA with a mixed-variable encoding scheme and adaptive hybrid-model learning (AHL) strategy.
• An orthogonal initialization (OI) strategy to manage large search spaces.
• A surrogate-assisted multi-level evaluation (SME) method to reduce computational expense, resulting in the SHEDA algorithm.

Main Results:

• SHEDA demonstrated high effectiveness and efficiency in hyperparameter optimization across benchmark datasets (CIFAR10, CIFAR100) and a real-world case study (aortic dissection diagnosis).
• Satisfactory hyperparameter configurations were achieved rapidly: 0.58 GPU days for CIFAR10, 0.97 for CIFAR100, and 1.18 for AD diagnosis.

Conclusions:

• The proposed SHEDA algorithm offers a significant advancement in efficient and effective hyperparameter optimization for CNNs.
• SHEDA successfully overcomes key challenges in hyperparameter tuning, providing a computationally feasible solution for complex deep learning models.