Strategies at a glance: A comparative analysis of training techniques for optimizing early-exit deep neural networks

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Early-exit deep neural networks (DNNs) allow adaptive inference, reducing computational costs by enabling predictions at intermediate layers.
• The performance of early-exit DNNs is significantly influenced by training strategies, which have not been systematically explored.

Purpose of the Study:

• To systematically compare six prominent training strategies for early-exit DNNs.
• To evaluate the impact of these strategies on accuracy, convergence, overfitting, and training efficiency across various architectures and datasets.

Main Methods:

• Compared Joint, Separate, Branch-wise, Two-stage, Distillation-based, and Hybrid strategies.
• Utilized MobileNet, ResNet, and VGG architectures on CIFAR-10, CIFAR-100, ImageNet-100, and ChestX-ray14 benchmarks.
• Employed statistical validation including ANOVA and Tukey's HSD tests.

Main Results:

• Joint and Distillation-based strategies showed strong generalization but higher computational costs.
• Two-stage and Branch-wise strategies exhibited overfitting at deeper exits; Separate training performed poorly at early exits.
• Hybrid strategies demonstrated a superior balance between accuracy and efficiency.

Conclusions:

• The choice of training strategy critically impacts early-exit DNN performance.
• Hybrid strategies provide optimal trade-offs for resource-constrained applications.
• Findings offer practical guidance for optimizing DNN training and inform future research.