对软max和RMSNorm的硬件导向近似,用于高效的变压器推理
Yiwen Kang1,2, Dong Wang1,2
1Institute of Information Science, Beijing Jiaotong University, Beijing 100044, China.
Micromachines
|January 28, 2026
概括
本研究介绍了硬件效率高的方法,通过优化非线性运算符 (如Softmax和RMSNorm) 来加速变压器推断. 这些技术可以降低资源成本和延迟,同时保持大型语言模型 (LLM) 的模型准确性.
科学领域:
- 计算机工程 计算机工程
- 人工智能的人工智能
- 软件工程 软件工程 软件工程
背景情况:
- 基于变压器的大型语言模型 (LLM) 在软件工程中越来越多地用于代码生成和NFR分类等任务.
- 现有的LLM优化研究主要针对线性运算,使得非线性运算符未被充分探索.
- 非线性运算符如Softmax和RMSNorm对于变压器性能至关重要,但在计算上昂贵.
研究的目的:
- 为变压器模型中的Softmax和RMSNorm操作员提出硬件高效的近似和加速方法.
- 为了降低资源成本并加快变压器推断速度.
- 为了保持LLMs的准确性,同时优化硬件利用率.
主要方法:
- 开发了一种带有范围缩小的SafeSoftmax技术,用于双方查找表 (LUT) 近似和加速.
- 使用帕雷托边界分析优化了比特宽度配置,并应用了对数值准确性的错误补偿.
- 用LOD驱动的LUT重构划分作为对数减去,并使用LOD进行并行计算优化RMSNorm.
主要成果:
- 实现了基于FPGA的管道加速器,证明了低操作员级延迟和低功耗.
- 在硬件资源使用方面实现了显著的减少.
- 尽管对Softmax和RMSNorm应用了近似和加速度,但模型的准确性仍然保持着.
结论:
- 提出的硬件效率高的方法通过优化关键的非线性运算符来有效地加速变压器推断.
- 基于FPGA的加速器提供了一个实用的解决方案,用于部署LLM,减少资源足迹和提高性能.
- 这项工作突出了非线性运算符的硬件级优化在推进LLM应用程序中的潜力.
相关概念视频
01:24Approximate Integration
46
In many practical and theoretical contexts, the exact value of a definite integral may be inaccessible. This limitation typically arises when the antiderivative of a function is either unknown or cannot be expressed in a closed mathematical form. Alternatively, it can occur when a function is defined not by a formula but by a finite set of empirical data points, such as those collected during experiments. In these cases, approximate integration techniques provide a valuable solution.One of the...
46
01:26Linearization and Approximation
57
Linearization is a mathematical technique used to approximate complex, nonlinear functions with simpler linear models in the vicinity of a chosen reference point. The method is based on the idea that, although a function may be difficult to evaluate exactly, its behavior near a specific input value can often be closely approximated by the tangent line at that point. This approach is particularly useful when small deviations from a known value are involved.Consider the square root function, for...
57
01:28Accuracy, limits, and approximation
1.3K
Accuracy, limits, and approximations are common in many fields, especially in engineering calculations. These concepts are imperative for ensuring that a given value is as close as possible to its true value.
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
1.3K
01:29Application of Linearization and Approximation
88
A drone flying through complex terrain often relies on more than one sensing method to estimate small changes in altitude. Along with direct measurements, air pressure provides a useful indirect indicator of vertical movement. Atmospheric pressure decreases as altitude increases, and this relationship is commonly described using an exponential model. Although accurate, converting pressure measurements into altitude values requires calculations that are too complex to perform repeatedly during...
88
01:33Bacterial Transformation
59.7K
In 1928, bacteriologist Frederick Griffith worked on a vaccine for pneumonia, which is caused by Streptococcus pneumoniae bacteria. Griffith studied two pneumonia strains in mice: one pathogenic and one non-pathogenic. Only the pathogenic strain killed host mice.
Griffith made an unexpected discovery when he killed the pathogenic strain and mixed its remains with the live, non-pathogenic strain. Not only did the mixture kill host mice, but it also contained living pathogenic bacteria that...
Griffith made an unexpected discovery when he killed the pathogenic strain and mixed its remains with the live, non-pathogenic strain. Not only did the mixture kill host mice, but it also contained living pathogenic bacteria that...
59.7K
01:26Linear Approximation in Frequency Domain
370
Linear systems are characterized by two main properties: superposition and homogeneity. Superposition allows the response to multiple inputs to be the sum of the responses to each individual input. Homogeneity ensures that scaling an input by a scalar results in the response being scaled by the same scalar.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
370