Hardware-Assisted Low-Latency NPU Virtualization Method for Multi-Sensor AI Systems.

Area of Science:

• Artificial Intelligence
• Computer Engineering
• Embedded Systems

Background:

• AI systems in autonomous driving and smart homes require processing complex text and image data.
• Current multi-sensor systems face challenges with low resource utilization and memory latency.
• Integrating NPUs and sensors improves speed but not always efficiency.

Purpose of the Study:

• To reduce processing time and enhance resource utilization in AI systems.
• To address challenges of low resource utilization and memory latency in NPU-based systems.
• To enable efficient multitasking and low-latency processing in resource-constrained environments.

Main Methods:

• Virtualizing neural processing units (NPUs) to handle multiple deep-learning models concurrently.
• Implementing a hardware scheduler to manage task execution and optimize resource allocation.
• Utilizing data prefetching techniques to minimize memory latency.

Main Results:

• Hardware scheduler reduced memory cycles by over 10% across tested models.
• Significant reductions in memory latency observed, e.g., 30% for NCF and 70% for DLRM.
• Minimized NPU idle time and memory latency, particularly in environments with frequent context switching.

Conclusions:

• The proposed virtualization method effectively improves NPU resource utilization and reduces processing latency.
• This approach is highly beneficial for real-time AI applications demanding efficient multitasking.
• Optimized performance in resource-constrained settings is achieved, supporting applications like autonomous driving and smart homes.