A Joint Model Provisioning and Request Dispatch Solution for Low-Latency Inference Services on Edge.

Area of Science:

• Computer Science
• Artificial Intelligence
• Edge Computing

Background:

• Mobile users increasingly depend on machine learning inference for critical decisions.
• Low-latency edge inference services are crucial for intelligent systems.
• Existing solutions for edge inference have limitations in optimizing resource utilization and request dispatching.

Purpose of the Study:

• To propose a novel solution for jointly provisioning machine learning models and dispatching inference requests.
• To reduce inference latency on edge nodes by optimizing resource allocation.
• To enhance the quality and responsiveness of edge inference services for mobile users.

Main Methods:

• Developed a solution that holistically considers networking, computing, and memory resources for provisioning edge nodes.
• Implemented the solution using TensorFlow Serving and Kubernetes on an edge cluster.
• Evaluated the strategy through simulations and testbed experiments under various system settings.

Main Results:

• The proposed joint strategy consistently achieved lower inference latency compared to traditional methods.
• Optimized provisioning of inference instances led to reduced serving latency for mobile users.
• Demonstrated the effectiveness of the holistic resource consideration in improving edge inference performance.

Conclusions:

• The joint provisioning and dispatching strategy offers a superior approach to reducing edge inference latency.
• This solution addresses the growing demand for high-quality, low-latency inference services in edge computing.
• The findings have significant implications for the development of responsive and efficient intelligent mobile applications.