Parareal Neural Networks Emulating a Parallel-in-Time Algorithm.

Area of Science:

• Artificial Intelligence
• Computer Science
• Computational Science

Background:

• Deep neural networks (DNNs) face increasing training times as model depth grows.
• Multi-GPU parallel computing is essential for accelerating DNN training.
• Existing parallelization methods often struggle with the sequential nature of DNN layers.

Purpose of the Study:

• To develop a novel methodology for constructing parallel neural networks that effectively utilize multiple GPUs.
• To accelerate the training of deep neural networks by parallelizing computations across layers.
• To maintain accuracy while achieving significant speedups in DNN training.

Main Methods:

• Interpreting DNN layers as time steps in a time-dependent problem.
• Emulating the parareal algorithm, a parallel-in-time approach, for DNN layer parallelization.
• Constructing a parallel structure by dividing DNN layers and connecting them with a coarse network.

Main Results:

• Demonstrated accelerated training for deep neural networks using the proposed methodology.
• Achieved accuracy preservation comparable to traditional sequential training.
• Successfully applied the method to established architectures like VGG-16 and ResNet-1001 on multiple datasets.

Conclusions:

• The proposed parareal emulation offers an effective strategy for multi-GPU parallelization of DNNs.
• This approach significantly reduces training time while preserving model accuracy.
• The methodology provides a viable solution for training increasingly deeper and more complex neural networks.