Distributed Newton Methods for Deep Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Optimization

Background:

• Deep learning training involves complex, non-convex optimization problems with numerous parameters.
• Distributed training is essential for large datasets and complex networks, but faces challenges with computational cost, communication, and synchronization.

Purpose of the Study:

• To propose a novel distributed Newton method for training deep neural networks with distributed model storage.
• To address the computational, communication, and synchronization bottlenecks in distributed deep learning.

Main Methods:

• Developed a distributed Newton method utilizing variable and feature-wise data partitions.
• Incorporated a diagonalization method to reduce communication costs for approximate Newton direction calculation.
• Employed subsampled Gauss-Newton matrices to decrease computation and communication overhead.
• Implemented a strategy to reduce synchronization costs by terminating early if some nodes are not finished.

Main Results:

• The proposed distributed Newton method effectively trains deep neural networks in distributed environments.
• The method demonstrates greater robustness and potentially higher test accuracy compared to stochastic gradient methods.
• Techniques for reducing running time and memory consumption were successfully integrated.

Conclusions:

• The novel distributed Newton method offers an effective solution for training deep neural networks with distributed model storage.
• The approach successfully mitigates communication and synchronization issues inherent in distributed training.
• Experimental results validate the method's effectiveness and superiority over stochastic gradient methods in certain aspects.