Design of Fully Spectral CNNs for Efficient FPGA-Based Acceleration

Area of Science:

• Computer Vision
• Deep Learning Hardware Acceleration
• Signal Processing

Background:

• Fast Fourier Transform (FFT)-based convolution reduces computational complexity in Convolutional Neural Networks (CNNs).
• Existing FFT-based methods face challenges with frequent spatial-frequency domain transformations, limiting low-latency inference, especially on embedded systems.
• The lack of nonlinear functions in the spectral domain hinders the efficiency of FFT-based CNNs.

Purpose of the Study:

• To propose a fully spectral CNN that removes compute-intensive domain transformations.
• To maintain the nonlinearity of spatial CNNs while enhancing hardware efficiency.
• To develop a customized hardware architecture for accelerating fully spectral CNN inference on FPGAs.

Main Methods:

• Introduced a novel spectral-domain adaptive rectified linear unit (ReLU) layer for fully spectral CNNs.
• Designed a compute-efficient hardware architecture with spectral-domain intralayer and interlayer pipeline techniques for FPGA acceleration.
• Employed a design space exploration (DSE) framework for optimizing resource allocation under hardware constraints.

Main Results:

• Achieved a throughput of 204 Giga Operations Per Second (Gop/s) with 80% compute efficiency on an Intel Arria 10 SX160 FPGA.
• Demonstrated 4x-6.6x speedup over state-of-the-art spatial CNN implementations.
• Showcased 3.0x-4.4x speedup over FFT-based implementations, maintaining similar accuracy.

Conclusions:

• The proposed fully spectral CNN with a novel spectral-domain ReLU layer effectively eliminates domain transformations, enhancing hardware efficiency.
• The customized FPGA accelerator significantly outperforms existing methods in throughput and efficiency for CNN inference.
• This approach offers a promising solution for low-latency, high-performance CNN inference on embedded platforms.