Zero time waste in pre-trained early exit neural networks

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Reducing processing time for large deep learning models is crucial for real-world applications.
• Early exit methods use internal classifiers (ICs) to reduce inference time by predicting easy examples early.
• Current methods waste computations from ICs that do not exit early.

Purpose of the Study:

• Introduce Zero Time Waste (ZTW), a novel approach to eliminate wasted computations in early exit methods.
• Improve the accuracy-inference time trade-off in deep learning models.

Main Methods:

• ZTW reuses predictions from preceding internal classifiers (ICs).
• Direct connections are added between ICs.
• Previous outputs are combined in an ensemble-like manner.

Main Results:

• ZTW demonstrates a superior accuracy vs. inference time trade-off compared to existing early exit methods.
• Achieved significant improvements on ImageNet, outperforming baselines in 11 out of 16 scenarios.
• Showcased up to 5 percentage points improvement on low computational budgets.

Conclusions:

• ZTW effectively reuses intermediate predictions, minimizing computational waste.
• The method offers a better balance between accuracy and inference speed for deep learning models.
• ZTW presents a promising solution for efficient processing of large deep learning models.