A Hierarchical Dispatcher for Scheduling Multiple Deep Neural Networks (DNNs) on Edge Devices

Area of Science:

• Computer Science
• Artificial Intelligence
• Embedded Systems

Background:

• Edge devices require efficient deep neural network (DNN) execution.
• Heterogeneous processing units (PUs) present scheduling challenges.
• Existing solutions lack scalability for diverse DNN workloads.

Purpose of the Study:

• To propose a hierarchical dispatcher architecture for efficient DNN scheduling on edge devices.
• To enable scalable and flexible workload management across heterogeneous PUs.
• To improve the performance of deep learning models deployed on edge systems.

Main Methods:

• Developed a hierarchical dispatcher with high-level and low-level components.
• Separated dispatcher functionality from scheduling policy.
• Designed for both integrated and distributed heterogeneous PUs.
• Formalized a hierarchical structure for managing DNN subgraphs.

Main Results:

• Achieved an average performance improvement of 51.6% in case studies.
• Demonstrated practical applicability on edge devices.
• Showcased efficient workload management in homogeneous and heterogeneous environments.
• Provided a scalable and adaptable solution for DNN deployment.

Conclusions:

• The hierarchical dispatcher architecture effectively schedules DNNs on heterogeneous edge devices.
• The proposed system offers significant performance gains and adaptability.
• This approach facilitates efficient deployment of deep learning models in edge computing.