Hierarchical Training of Deep Neural Networks Using Early Exiting

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) achieve high accuracy in vision tasks but require substantial computational resources for training.
• Training DNNs on remote cloud servers incurs high communication costs, extended runtimes, and privacy risks due to data transfer from edge devices.
• Existing methods struggle with simultaneous hierarchical training or compromise data privacy.

Purpose of the Study:

• To propose a novel hierarchical training method for DNNs that mitigates communication costs, training runtime, and privacy concerns.
• To enable simultaneous training across edge and cloud workers without sharing raw input data.
• To leverage early exits for a divided backward pass during DNN training.

Main Methods:

• A hierarchical training framework is introduced, dividing DNN architecture between edge and cloud.
• Early exits are utilized to partition the backward pass, enabling parallel processing between edge and cloud.
• The method ensures raw data privacy by not transmitting it to the cloud and eliminates communication during the backward pass.

Main Results:

• The proposed method demonstrated significant reductions in training runtime for VGG-16 (29-25%) and ResNet-18 (61-81%) on CIFAR-10 and Tiny ImageNet datasets, respectively.
• These runtime improvements were achieved over low bit rate channels with negligible impact on model accuracy.
• Simulations and on-device experiments validated the effectiveness across various DNN architectures.

Conclusions:

• The novel hierarchical training method offers an efficient solution for training high-accuracy DNNs on resource-constrained edge devices within edge-cloud systems.
• This approach enhances the flexibility of edge devices for online learning, enabling adaptation to new tasks and data classes.
• The method addresses key challenges in distributed DNN training, improving efficiency and privacy.