Spartus: A 9.4 TOp/s FPGA-Based LSTM Accelerator Exploiting Spatio-Temporal Sparsity

Area of Science:

• Computer Science
• Artificial Intelligence
• Hardware Acceleration

Background:

• Long Short-Term Memory (LSTM) networks are crucial for processing time-sequential data like speech.
• Existing LSTM accelerators focus on either spatial or temporal sparsity, limiting efficiency.
• There is a need for accelerators that leverage both spatial and temporal sparsity for improved performance.

Purpose of the Study:

• To introduce Spartus, a new accelerator designed for ultralow latency inference in LSTM networks.
• To exploit spatio-temporal sparsity in LSTMs for enhanced efficiency.
• To demonstrate real-time online speech recognition capabilities.

Main Methods:

• Implemented a column-balanced targeted dropout (CBTD) method to induce structured spatial sparsity in LSTM weights.
• Extended the DeltaGRU method to DeltaLSTM to induce temporal sparsity.
• Developed the Spartus hardware architecture to process these sparse LSTM networks.

Main Results:

• Achieved high weight sparsity levels (up to 96% and 94%) with negligible accuracy loss on TIMIT and Librispeech datasets.
• Demonstrated ultralow latency with an average per-sample latency of 1 μs for a single DeltaLSTM layer.
• Achieved a 46x speedup over theoretical performance, resulting in 9.4-TOp/s throughput and 1.1-TOp/s/W power efficiency.

Conclusions:

• Spartus effectively exploits spatio-temporal sparsity for ultralow latency LSTM inference.
• The architecture supports scalable, real-time online speech recognition on FPGAs.
• This approach offers significant improvements in speed and power efficiency for AI hardware acceleration.