EdgeOpt-Sched-CS: Cold-Start-Aware Dynamic Scheduling for Efficient DNN Inference at the Edge

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Dynamic scheduling enhances deep neural network (DNN) inference efficiency on edge devices.
• Cold-start overhead occurs during initial model deployment, requiring online profiling and adaptation.
• Existing dynamic scheduling methods lack effective strategies for mitigating cold-start issues.

Purpose of the Study:

• To propose EdgeOpt-Sched-CS, a novel framework for cold-start-aware dynamic graph scheduling for edge inference.
• To leverage knowledge transfer from similar computation graphs to initialize schedulers.
• To reduce the latency and improve the stability of DNN models during their early deployment phases on edge devices.

Main Methods:

• Developed EdgeOpt-Sched-CS, an extension of dynamic graph scheduling.
• Implemented a knowledge transfer mechanism using compact graph signatures to retrieve relevant source schedulers.
• Incorporated lightweight, cold-start-aware online adaptation for early deployment phases.
• Evaluated the framework on diverse edge device-model scenarios, including CNNs, transformers, and quantized language models.

Main Results:

• EdgeOpt-Sched-CS reduced cumulative cold-start latency by 10.6-20.4%.
• The framework shortened the time-to-stability by 5.2-21.7%.
• Steady-state latency and memory efficiency were maintained with minimal additional scheduling overhead.

Conclusions:

• Scheduler initialization using prior knowledge from similar graphs is a crucial optimization for adaptive edge inference.
• EdgeOpt-Sched-CS effectively reuses scheduling knowledge across related computation graphs, significantly mitigating cold-start problems.
• The proposed method offers a practical solution for deploying DNNs efficiently on edge devices with reduced initial performance degradation.