Spectral Tensor Layers for Communication-Free Distributed Deep Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Distributed deep learning often requires significant communication overhead.
• Handling heterogeneous data, especially with varying resolutions, presents a challenge in federated learning.

Purpose of the Study:

• To propose a novel spectral tensor layer for communication-free distributed deep learning.
• To enable efficient training on heterogeneous datasets without inter-node communication.

Main Methods:

• Representing data in tensor form and replacing matrix products with tensor products.
• Splitting datasets into spectral subdatasets using linear transforms for parallel processing.
• Ensembling parallel network branches trained on subdatasets to form the final network.

Main Results:

• Achieved communication-free distributed learning across multiple datasets (MNIST, CIFAR-10, ImageNet).
• Demonstrated reduced memory and computation costs compared to traditional networks.
• Showcased effective learning with multiresolution heterogeneous data.

Conclusions:

• The spectral tensor network offers a communication-free, efficient solution for distributed deep learning.
• The method provides benefits in generalization, storage reduction, and parallel computation speedup.
• It is a promising approach for federated learning with diverse data resolutions.