Hardware-Oriented Approximations of Softmax and RMSNorm for Efficient Transformer Inference.

Area of Science:

• Computer Engineering
• Artificial Intelligence
• Software Engineering

Background:

• Transformer-based large language models (LLMs) are increasingly used in software engineering for tasks like code generation and NFR classification.
• Existing research on LLM optimization primarily targets linear operations, leaving nonlinear operators underexplored.
• Nonlinear operators such as Softmax and RMSNorm are critical for Transformer performance but are computationally expensive.

Purpose of the Study:

• To propose hardware-efficient approximation and acceleration methods for Softmax and RMSNorm operators in Transformer models.
• To reduce resource costs and accelerate Transformer inference speed.
• To maintain the accuracy of LLMs while optimizing hardware utilization.

Main Methods:

• Developed a SafeSoftmax technique with range reduction for bipartite lookup table (LUT) approximation and acceleration.
• Optimized bit-width configuration using Pareto frontier analysis and applied error compensation for numerical accuracy.
• Reformulated division as logarithmic subtraction using a LOD-driven LUT and optimized RMSNorm using LOD for parallel computation.

Main Results:

• Implemented an FPGA-based pipelined accelerator demonstrating low operator-level latency and power consumption.
• Achieved significant reductions in hardware resource usage.
• Preserved model accuracy despite the approximations and accelerations applied to Softmax and RMSNorm.

Conclusions:

• The proposed hardware-efficient methods effectively accelerate Transformer inference by optimizing critical nonlinear operators.
• The FPGA-based accelerator offers a practical solution for deploying LLMs with reduced resource footprints and improved performance.
• This work highlights the potential of hardware-level optimizations for nonlinear operators in advancing LLM applications.