Corun: Concurrent Inference and Continuous Training at the Edge for Cost-Efficient AI-Based Mobile Image Sensing.

Area of Science:

• Computer Science
• Artificial Intelligence
• Edge Computing

Background:

• Deep learning-powered mobile image sensing faces challenges with limited device resources, causing high latency and battery drain.
• Inference accuracy can degrade over time due to data drift, necessitating model updates.
• Current edge deployments often require separate GPUs for inference and retraining, increasing costs.

Purpose of the Study:

• To introduce Corun, a cost-efficient framework for simultaneous deep learning inference and model retraining on a single edge GPU.
• To improve inference throughput and maintain accuracy for mobile image sensing applications.
• To reduce the hardware costs associated with edge AI deployments.

Main Methods:

• Corun utilizes offline profiling to determine the optimal number of concurrent inference queries that can run with retraining on one GPU.
• It employs a scheduling method to manage multiple inference tasks and continual model retraining without causing out-of-memory errors or significant latency increases.
• The framework focuses on a single commodity GPU in an edge server.

Main Results:

• Corun significantly enhances inference throughput, scaling with the number of concurrent inference queries.
• Latency and retraining epoch length increase at substantially lower rates compared to separate task processing.
• Evaluations confirm the cost-effectiveness of Corun in reducing GPU requirements and deployment costs.

Conclusions:

• Corun offers a practical solution for resource-constrained edge environments, enabling efficient deep learning-based mobile image sensing.
• By consolidating inference and retraining, Corun demonstrates a viable strategy for cost reduction in edge AI.
• The framework successfully addresses latency, battery consumption, and accuracy maintenance challenges in mobile deep learning applications.