NUPES:通过功率指数搜索进行非统一的训练后量化
概括
本研究介绍了NUPES,这是一种用于深度神经网络 (DNN) 和大型语言模型 (LLM) 的新型非统一量子化方法. 在训练过程中,NUPES优化了量化参数,实现了最先进的压缩速率,以实现高效的模型部署.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 计算机工程 计算机工程
背景情况:
- 深度神经网络 (DNN) 的部署受到高计算成本的限制,特别是在大型语言模型 (LLM) 中.
- 量化,将浮点转换为固定点表示,减少了内存和延迟,但统一的方法与非钟形DNN重量/激活分布和LLM异常值相斗争.
- 现有的训练后量子化技术不足以有效地优化指数和权重等量子化参数.
研究的目的:
- 提出NUPES,一种先进的非统一量子化技术,以克服DNN和LLMs中统一量子化的局限性.
- 开发一种新的训练范式,以优化整个量子化空间内的量子化运算符和权重.
- 为了实现仅整数,低位推断,同时保持模型性能和实现高压缩速率.
主要方法:
- 在量子化过程中,NUPES利用功率函数衍生的自律形态来保持标量乘法.
- 一个新的训练范式学习了整个量子化空间中的量子化权重,并优化了量子化运算符的指数参数.
- 数字不稳定性得到缓解,使量子化过程的端到端训练成为可能.
主要成果:
- 在无数据和数据驱动的量子化配置中,NUPES实现了最先进的压缩速率.
- 该方法有效地解决了对具有异常值的分布的统一量子化的局限性,特别是在变压器和LLM中.
- 经验性基准表明,与以前的培训后量化技术相比,NUPES的性能优越.
结论:
- 在非均量子化方面,NUPES为高效的DNN和LLM部署提供了显著的进展.
- 拟议的训练范式成功优化了量化参数,从而实现了卓越的压缩和性能.
- NUPES提供了一个可行的解决方案,用于在资源有限的硬件上部署大型模型.
相关概念视频
01:22Upsampling
311
Managing signal sampling rates is essential in digital signal processing to maintain signal integrity. A decimated signal, characterized by a reduced frequency range due to its lower sampling rate, can be upsampled by inserting zeros between each sample. This upsampling process expands the original spectrum and introduces repeated spectral replicas at intervals dictated by the new Nyquist frequency. To refine this zero-inserted sequence, it is passed through a lowpass filter with a cutoff...
311
01:26Linear Approximation in Frequency Domain
131
Linear systems are characterized by two main properties: superposition and homogeneity. Superposition allows the response to multiple inputs to be the sum of the responses to each individual input. Homogeneity ensures that scaling an input by a scalar results in the response being scaled by the same scalar.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
131
01:18Parseval's Theorem
653
Parseval's theorem is a fundamental concept in signal processing and harmonic analysis. It asserts that for a periodic function, the average power of the signal over one period equals the sum of the squared magnitudes of all its complex Fourier coefficients. This theorem, named after Marc-Antoine Parseval, provides a powerful tool for analyzing the energy distribution in signals.
Interestingly, Parseval's theorem also holds for the trigonometric form of the Fourier series, which...
Interestingly, Parseval's theorem also holds for the trigonometric form of the Fourier series, which...
653
01:21Linear Approximation in Time Domain
125
Nonlinear systems often require sophisticated approaches for accurate modeling and analysis, with state-space representation being particularly effective. This method is especially useful for systems where variables and parameters vary with time or operating conditions, such as in a simple pendulum or a translational mechanical system with nonlinear springs.
For a simple pendulum with a mass evenly distributed along its length and the center of mass located at half the pendulum's length,...
For a simple pendulum with a mass evenly distributed along its length and the center of mass located at half the pendulum's length,...
125
01:15Sampling Theorem
766
In signal processing, the analysis of continuous-time signals, denoted as x(t), often involves sampling techniques to convert these signals into discrete-time signals. This process is essential for digital representation and manipulation. A critical component in sampling is the train of impulses, characterized by the sampling interval and the sampling frequency. The relationship between these parameters and the original signal's properties dictates the success of the sampling process.
766
01:11Sampling Continuous Time Signal
350
In signal processing, a continuous-time signal can be sampled using an impulse-train sampling technique, followed by the zero-order hold method. Impulse-train sampling involves the use of a periodic impulse train, which consists of a series of delta functions spaced at regular intervals determined by the sampling period. When a continuous-time signal is multiplied by this impulse train, it generates impulses with amplitudes corresponding to the signal's values at the sampling points.
In the...
In the...
350