Related Experiment Video
Updated: Oct 17, 2025
05:41
A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis
Published on: February 6, 2020
9.6K
Deploying and scaling distributed parallel deep neural networks on the Tianhe-3 prototype system.
Jia Wei1, Xingjun Zhang2, Zeyu Ji1
1Xi'an Jiaotong University, Xi'an, 710049, Shaanxi, China.
Scientific Reports
|October 13, 2021
Summary
Deep learning training on the Tianhe-3 supercomputer is accelerated using an optimized gradient synchronization strategy. This research enhances deep neural network (DNN) training performance on ARM-based architectures.
Area of Science:
- Computer Science
- Artificial Intelligence
- High-Performance Computing
Background:
- Increasing computing power enables deeper and more complex Deep Neural Networks (DNNs), improving feature extraction and data fitting.
- However, large datasets and complex models significantly increase DNN training overhead, necessitating faster training processes.
- The Tianhe-3 supercomputer's E-class target speed offers substantial computational power for DNN training.
Purpose of the Study:
- To accelerate the training process of Deep Neural Networks (DNNs) on the Tianhe-3 supercomputer.
- To implement and extend popular DNN models (LeNet, AlexNet, VGG, ResNet) on Tianhe-3's compute nodes.
- To propose and validate an optimized gradient synchronization strategy for large-scale distributed DNN training.
Main Methods:
- Implemented and extended LeNet, AlexNet, VGG, and ResNet model training on single MT-2000+ and FT-2000+ compute nodes.
- Extended training to multi-node clusters on the Tianhe-3 prototype.
- Developed and proposed an improved gradient synchronization process, a Dynamic Allreduce communication optimization strategy, tailored for ARM architecture features.
Main Results:
- Demonstrated the feasibility of training various DNN models on Tianhe-3's compute nodes and clusters.
- The proposed Dynamic Allreduce strategy showed potential for optimizing gradient synchronization.
- Provided experimental data supporting the enhancement of Tianhe-3's performance in large-scale distributed DNN training.
Conclusions:
- The Tianhe-3 supercomputer, with its significant computing power, can be effectively utilized for accelerating DNN training.
- The developed gradient synchronization optimization strategy offers a theoretical and experimental basis for improving distributed DNN training performance on ARM architectures.
- Further enhancements to the Tianhe-3 prototype can be guided by these findings for large-scale neural network training.
Related Concept Videos
01:20Parallel Processing
357
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
357
01:21Distributed Loads: Problem Solving
821
Beams are structural elements commonly employed in engineering applications requiring different load-carrying capacities. The first step in analyzing a beam under a distributed load is to simplify the problem by dividing the load into smaller regions, which allows one to consider each region separately and calculate the magnitude of the equivalent resultant load acting on each portion of the beam. The magnitude of the equivalent resultant load for each region can be determined by calculating...
821