Hybrid Training Method for MLP: Optimization of Architecture and Training

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Science

Background:

• Artificial neural network (ANN) performance is critically dependent on optimal connection weights, network architecture, and cost function selection during training.
• Existing optimization methods may face challenges in efficiently searching complex solution spaces for ANNs.

Purpose of the Study:

• To present GaTSa, a novel hybrid approach for optimizing both the architecture and weights of artificial neural networks (ANNs).
• To extend the authors' previous TSa method by integrating genetic algorithms (GA) for enhanced ANN performance optimization.

Main Methods:

• GaTSa integrates simulated annealing (SA), tabu search (TS), genetic algorithms (GA), and backpropagation for hybrid optimization.
• A constructive process using GA adds new nodes to the ANN architecture.
• SA enables escaping local minima via uphill moves, while TS improves convergence by evaluating solution sets.

Main Results:

• Empirical evaluation on 11 public-domain datasets demonstrated GaTSa's effectiveness in simultaneously optimizing ANN architecture and weights.
• GaTSa showed capability for relevant feature selection.
• Statistically significant improvements were observed compared to other global and local optimization techniques.

Conclusions:

• GaTSa offers a robust and effective hybrid optimization strategy for artificial neural networks.
• The method successfully addresses challenges in ANN training by optimizing architecture, weights, and enabling feature selection.