Enhancing Distributed Neural Network Training Through Node-Based Communications.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Modern deep neural networks (DNNs) require vast amounts of data, leading to substantial computational demands.
• Data parallelism strategies, while improving runtimes, face bottlenecks due to extensive gradient communication.
• Communication latency in distributed platforms significantly impacts the efficiency of DNN training.

Purpose of the Study:

• To develop and present node-based optimization steps for reducing gradient exchange in DNNs.
• To create a versatile communication scheme applicable to various general-purpose DNN algorithms.
• To address communication latency issues hindering performance on distributed platforms.

Main Methods:

• Implemented node-based optimization to minimize gradient exchange between model replicas.
• Developed a communication scheme considering the location of each replica within the distributed platform.
• Evaluated the proposal using diverse neural network architectures, datasets, and application types.

Main Results:

• Demonstrated significant reduction in global training time for deep neural networks.
• Achieved a slight improvement in model accuracy.
• Validated the robustness and versatility of the proposed communication scheme across different scenarios.

Conclusions:

• The proposed node-based optimization effectively reduces gradient communication bottlenecks in DNN training.
• The versatile communication scheme enhances efficiency and accuracy in distributed deep learning.
• This approach offers a practical solution for improving the performance of large-scale DNNs.