EQ-ViT: Algorithm-Hardware Co-Design for End-to-End Acceleration of Real-Time Vision Transformer Inference on Versal...

Area of Science:

• Computer Vision
• Hardware Acceleration
• Machine Learning

Background:

• Vision Transformers (ViTs) show promise in computer vision but face deployment challenges for real-time (< 1 ms) applications.
• Existing platforms (CPUs, GPUs, FPGAs) struggle with deterministic low-latency requirements, even with model quantization.
• Pruning and sparsity techniques reduce model size but often lead to accuracy loss.

Purpose of the Study:

• To propose EQ-ViT, an end-to-end acceleration framework for real-time ViT deployment.
• To co-design algorithms and hardware architectures for efficient ViT acceleration on AMD Versal ACAP.
• To overcome the accuracy-latency trade-off in current ViT acceleration methods.

Main Methods:

• In-depth kernel-level performance profiling to identify bottlenecks in existing acceleration solutions.
• Development of a spatial and heterogeneous EQ-ViT architecture leveraging ACAP's FPGA and AI Engine (AIE) resources.
• Implementation of a quantization-aware training strategy (EQ-ViT algorithm) for 8-bit weight and activation quantization, including nonlinear functions.
• Design of an automation framework to deploy EQ-ViT for various ViT applications on AMD Versal ACAP.

Main Results:

• Achieved a 2.4% accuracy improvement with EQ-ViT.
• Obtained average speedups of 315.0x over vCPUs, 3.39x-59.5x over GPUs, and 3.38x-13.1x over FPGAs.
• Demonstrated significant energy efficiency gains, ranging from 12.82x to 62.2x compared to various computing solutions.

Conclusions:

• EQ-ViT effectively enables real-time Vision Transformer acceleration on AMD Versal ACAP.
• The proposed framework achieves superior performance and energy efficiency without compromising accuracy.
• EQ-ViT offers a viable solution for deploying demanding computer vision tasks in real-time scenarios.