Learned Gradient Compression for Distributed Deep Learning.

Area of Science:

• Machine Learning
• Computer Science
• Artificial Intelligence

Background:

• Deep neural network training demands substantial computation, especially with high-dimensional data.
• Data-parallel distributed training addresses this by replicating models across nodes with data subsets.
• High communication rates and latency hinder distributed training, particularly over wireless networks.

Purpose of the Study:

• To reduce communication costs in distributed deep learning.
• To improve gradient compression efficiency by leveraging inter-node redundancy.
• To propose novel gradient compression methods suitable for different communication protocols.

Main Methods:

• Developed Learned Gradient Compression (LGC) methods, utilizing an autoencoder to capture common gradient information across nodes.
• Implemented a lightweight neural network for the autoencoder to manage computational complexity.
• Tested LGC on image classification and semantic segmentation tasks with various CNNs and datasets.

Main Results:

• Achieved significant compression rate reductions, e.g., 8095× over baseline and 8× over DGC for ResNet101 on Cifar10.
• Maintained high accuracy (93.57%) for ResNet101 on Cifar10, with only a 0.18% drop from the baseline.
• Demonstrated effectiveness across different CNN architectures and datasets.

Conclusions:

• LGC effectively leverages inter-node gradient correlation for superior compression efficiency.
• The proposed methods offer a practical solution for reducing communication bottlenecks in distributed deep learning.
• LGC provides a significant advancement over existing gradient compression techniques.