Multi-Model Running Latency Optimization in an Edge Computing Paradigm.

Area of Science:

• Artificial Intelligence
• Edge Computing
• Software Engineering

Background:

• Lightweight deep learning and edge computing enable concurrent multi-model inference on resource-constrained devices.
• High latency in multi-model inference hinders real-time applications.
• Optimization is needed to minimize latency without compromising safety-critical performance.

Purpose of the Study:

• To develop a real-time task scheduling strategy for multi-model deployment on edge devices.
• To investigate model inference using the Open Neural Network Exchange (ONNX) runtime.
• To propose an application deployment strategy using container technology for optimized inference task scheduling.

Main Methods:

• Developed a real-time task scheduling strategy.
• Utilized the Open Neural Network Exchange (ONNX) runtime engine for model inference.
• Implemented an application deployment strategy using container technology.
• Scheduled inference tasks to different containers based on developed strategies.

Main Results:

• The proposed solution significantly reduces overall running latency for multi-model inferences.
• The strategy effectively manages concurrent inference tasks on edge devices.
• Containerization facilitated efficient scheduling and deployment of inference tasks.

Conclusions:

• The developed task scheduling strategy is effective in reducing latency for real-time multi-model inference on edge devices.
• Container technology provides a viable framework for deploying and managing inference tasks.
• This approach enables collaborative goal achievement through efficient multi-model deployment.