Fast parallel off-line training of multilayer perceptrons

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Science

Background:

• Multilayer perceptron (MLP) training algorithms, particularly second-order gradient-based methods, are computationally intensive.
• Efficient parallel implementations are crucial for handling large-scale machine learning problems.

Purpose of the Study:

• To describe and evaluate various parallel implementation strategies for second-order gradient-based MLP training.
• To compare the performance of Hessian-based and conjugate gradient-based methods on different parallel architectures.

Main Methods:

• Focus on limited- and full-memory Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithms as representative examples.
• Investigate parallelization strategies, implementing the most effective on Parallel Virtual Machine (PVM) and transputer architectures.
• Analyze the impact of step size, gradient calculations, and matrix operations on performance.

Main Results:

• Transputer-based implementations achieve excellent speed-ups for MLP training, but are constrained by memory limitations.
• Parallel Virtual Machine (PVM) implementations exhibit significantly poorer speed-ups due to inefficient communication, though they do not face memory constraints.
• Step size and gradient calculations are identified as critical components affecting algorithm performance.

Conclusions:

• The choice of parallel architecture significantly impacts the performance and scalability of MLP training algorithms.
• Transputer systems offer high performance for smaller datasets, while PVM may be more suitable for larger problems if communication overhead can be managed.
• Further research into optimizing communication protocols for PVM could enhance its efficiency for parallel gradient-based training.