Analysis of training set parallelism for backpropagation neural networks

Area of Science:

• Computer Science
• Artificial Intelligence
• Parallel Computing

Background:

• Feedforward neural networks are commonly parallelized using training set or network-based parallelism.
• Training set parallelism is effective for large datasets relative to network size, especially with backpropagation.

Purpose of the Study:

• To analyze training set parallelism for feedforward neural networks on a pipelined ring transputer array.
• To derive theoretical expressions for performance metrics and optimal network configurations.
• To investigate optimal training data distribution strategies.

Main Methods:

• Theoretical analysis of training set parallelism on a transputer array in a pipelined ring topology.
• Derivation of expressions for time per epoch and optimal processor network size.
• Investigation of training data distribution, including non-integer multiples of processors.
• Extension of analysis to heterogeneous processor networks.
• Experimental verification using a T805 transputer array.

Main Results:

• Speedup is dependent on patterns per processor, communication overhead, and network size.
• Equal distribution of training data is not always optimal, even with integer multiples.
• Optimal allocation can be a mixed integer programming problem.
• Theoretical results are validated by experimental data.

Conclusions:

• The optimal distribution of training data significantly impacts the efficiency of parallel feedforward neural networks.
• The findings provide insights into optimizing parallel processing for neural networks on specific hardware architectures.
• The study validates theoretical models with practical experimental results.