Efficient Layer-Wise N:M Sparse CNN Accelerator with Flexible SPEC: Sparse Processing Element Clusters

Area of Science:

• Computer Engineering
• Artificial Intelligence
• Hardware Acceleration

Background:

• Layer-wise N:M fine-grained sparsity reduces computational complexity in neural networks with minimal accuracy loss.
• Existing hardware accelerators often fail to fully exploit the speed-up potential of N:M sparsity.
• Efficient hardware support is crucial for realizing the benefits of sparse neural network algorithms.

Purpose of the Study:

• To design an efficient hardware accelerator specifically for N:M sparse convolutional neural networks (CNNs) with layer-wise sparse patterns.
• To analyze and select optimal processing element (PE) structures for flexible PE architecture.
• To incorporate variable sparse convolutional dimensions and sparse ratios into the hardware design.

Main Methods:

• Developed a flexible processing element (PE) architecture by analyzing different PE structures.
• Designed a Sparse PE Cluster (SPEC) to efficiently handle layer-wise N:M sparse patterns.
• Integrated the SPEC into a CNN accelerator featuring flexible network-on-chip and specialized dataflow.
• Implemented hardware accelerators on Xilinx ZCU102 and VCU118 FPGAs.

Main Results:

• The proposed accelerator efficiently handles N:M sparse patterns in CNNs.
• Evaluated performance on Alexnet, VGG-16, and ResNet-50.
• Achieved superior power efficiency compared to existing accelerators for structured and unstructured pruned networks.

Conclusions:

• The designed hardware accelerator effectively supports layer-wise N:M sparse CNNs.
• The SPEC design and integrated dataflow enable efficient acceleration of sparse CNNs.
• The proposed solution offers significant power efficiency advantages for sparse CNN hardware acceleration.