Local Critic Training for Model-Parallel Learning of Deep Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Training deep neural networks, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs), often requires significant computational resources.
• Existing parallel learning methods face challenges in efficiently decoupling the update processes of different network layers.

Purpose of the Study:

• To introduce a novel model-parallel learning method, local critic training, for efficient neural network training.
• To demonstrate the ability of local critic training to decouple layer group updates in CNNs and RNNs.
• To validate the convergence properties and applicability of the trained networks for structural optimization.

Main Methods:

• The proposed method, local critic training, utilizes auxiliary local critic networks to estimate error gradients.
• The main neural network is partitioned into layer groups, with each group updated via gradients from its corresponding local critic network.
• The method was evaluated on both CNN and RNN architectures.

Main Results:

• Local critic training successfully decouples the update process for layer groups in both CNNs and RNNs.
• The method is theoretically guaranteed to converge to a critical point.
• Networks trained with this method are suitable for structural optimization tasks.
• Experimental results indicate satisfactory performance, significant reductions in training time, and decreased memory consumption per machine.

Conclusions:

• Local critic training offers an effective and efficient approach to model-parallel learning.
• The method provides a robust framework for training deep neural networks with improved resource utilization.
• The successful application in structural optimization highlights the versatility of the proposed technique.