ESCAN: Efficient GPU sharing for cascade neural network inference.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cascading models balance efficiency and accuracy in industrial deployments.
• Low-latency services demand optimized execution efficiency and device utilization.
• Existing GPU sharing (Multiprocessing Services) struggles with cascade models due to early-exit and execution order.

Purpose of the Study:

• To address the challenges of applying GPU sharing to cascade neural networks.
• To propose a framework that optimizes inference efficiency for cascade models.
• To enhance device utilization and reduce latency in cloud-based inference services.

Main Methods:

• Analyzed cascade neural network characteristics and device-sharing techniques.
• Developed ESCAN, a GPU-sharing optimization framework for online inference.
• Integrated exit-ratio-aware batch-parallel execution and resource allocation algorithms in PyTorch.

Main Results:

• ESCAN improves inference efficiency by an average of 19.53% compared to parallel execution.
• Significantly enhances the efficiency of searching for computation resource allocation schemes.
• Optimizes computational resource utilization through effective GPU-sharing.

Conclusions:

• ESCAN provides an effective solution for GPU-sharing optimization in cascade neural networks.
• Achieves a balance between device-sharing gains and early-exit computation wastage.
• Delivers a low-latency, high-precision optimization for interactive online services using cascade models.