A Heterogeneous Architecture for the Vision Processing Unit with a Hybrid Deep Neural Network Accelerator

Area of Science:

• Computer Engineering
• Artificial Intelligence
• Image Processing

Background:

• Vision chips integrate image sensors and processing units for vision tasks.
• Modern vision tasks require both image signal processing (ISP) algorithms and deep neural networks (DNNs).
• Existing vision processing units (VPUs) and deep neural network processing units (DNPUs) have limitations in handling diverse vision tasks.

Purpose of the Study:

• To propose a heterogeneous VPU architecture with a hybrid accelerator capable of processing ISP and DNN tasks.
• To develop a resource-sharing scheme for multiplexing hardware among different subtasks.
• To enhance processing speed and efficiency for vision tasks.

Main Methods:

• Designed a heterogeneous VPU architecture incorporating a hybrid accelerator for DNNs.
• Implemented a hardware resource sharing scheme for multiplexing subtasks.
• Utilized a pipelined workflow to optimize the use of processing modules.
• Implemented the VPU on a field-programmable gate array (FPGA) for testing.

Main Results:

• The proposed VPU can process ISP, CNNs, and hybrid DNN subtasks on a single unit.
• The sharing scheme effectively multiplexes hardware resources.
• The pipelined workflow ensures full utilization of processing modules.
• Achieved an average performance of 22.6 giga operations per second per Watt (GOPS/W) on tested vision tasks.

Conclusions:

• The developed heterogeneous VPU architecture offers an efficient solution for modern vision tasks.
• The hybrid accelerator and resource-sharing scheme enable efficient processing of diverse algorithms.
• The FPGA implementation demonstrates the practical viability and high performance of the proposed design.