Hybrid interior point training of modular neural networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Modular neural networks (MNNs) utilize gating neurons to integrate outputs from multiple agent neurons.
• Expectation-Maximization (EM) algorithms are a common approach for training MNNs to approximate complex non-linear functions.
• Existing training methods for MNNs can be computationally intensive and may face challenges in achieving optimal performance.

Purpose of the Study:

• Introduce a novel Hybrid Interior-Point (HIP) algorithm for training modular neural networks.
• Evaluate the computational efficiency and approximation accuracy of the HIP algorithm.
• Compare the performance of the HIP algorithm against established Expectation-Maximization (EM) methods.

Main Methods:

• Developed a Hybrid Interior-Point (HIP) algorithm combining interior-point linear programming (LP) with Newton-Raphson iteration.
• Formally proved the asymptotic convergence of the HIP algorithm to locally optimal MNNs.
• Conducted simulation experiments to assess approximation error and computational scaling.

Main Results:

• The HIP algorithm demonstrated superior average approximation error compared to EM-trained networks.
• HIP algorithm's computational cost exhibits polynomial scaling with problem size.
• HIP algorithm shows a slower computational cost scaling rate than the EM procedure, with comparable costs for small networks.

Conclusions:

• The HIP algorithm offers an efficient and effective method for training modular neural networks.
• HIP provides a computationally advantageous alternative to EM algorithms, especially for larger problems.
• This research advances MNN training methodologies, promising improved performance and scalability.