Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Optimizing deep neural network (DNN) architecture and hyperparameters is crucial for performance.
• Manual hyperparameter tuning is time-consuming and suboptimal.
• Existing automatic methods may lack efficiency or effectiveness in exploring the search space.

Purpose of the Study:

• To propose a novel automatic hyperparameter selection approach for DNNs.
• To combine particle swarm optimization (PSO) with steepest gradient descent for efficient network configuration.
• To improve the generalization performance of DNNs through optimized network structures and hyperparameters.

Main Methods:

• Network configurations encoded as real-number vectors for PSO individuals.
• PSO algorithm for global search of optimal network configurations.
• Steepest gradient descent for local training of DNN classifiers during PSO evaluation and final model training.

Main Results:

• The proposed hybrid approach effectively determined optimal network structures and hyperparameters.
• Experiments on handwritten characters and biological activity prediction showed superior generalization performance.
• DNN classifiers trained with the proposed method outperformed those from a random approach.

Conclusions:

• The combined PSO and steepest gradient descent method offers an effective tool for automatic DNN optimization.
• This approach provides a viable alternative for selecting network structures and hyperparameters.
• The method demonstrates improved generalization capabilities for DNN models.