LRQuant:基于变压器的大型基础模型的培训后量化统一和可学习的框架
IEEE transactions on pattern analysis and machine intelligence
|August 14, 2025
概括
本研究介绍了LRQuant,这是一种用于大型基础模型的新型训练后量化方法. LRQuant优化了缩放因子,并使用了一个新的损失函数来改善各种场景中的模型效率和准确性.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 计算机视觉 计算机视觉
背景情况:
- 训练后量化 (PTQ) 加快了推断,并减少了大型基础模型 (LFMs) 的记忆,而不需要重新训练.
- 现有的PTQ方法与手工制作的缩放因子扎,忽略定向定量化错误,缺乏广泛的应用.
- 当前的量化误差指标 (例如L2-norm) 不能捕捉方向变化,导致性能不足最佳.
研究的目的:
- 为基于变压器的LFM开发一个统一,可学习和强大的培训后量化框架 (LRQuant).
- 通过引入可学习的缩放因子和一种新的损失函数来解决现有的PTQ方法的局限性.
- 提供各种LFM和量子化场景的全面评估,包括具有挑战性的低位设置.
主要方法:
- 引入了一个区块智能可学习的范式,用于最佳的缩放因子确定,初始化为对数激活等价值.
- 提出了一种新的负对数共弦相似度 (NLC) 损失,以更好地捕捉超出MSE的量子化错误.
- 开发了具有动态损失权重方案的LRQuant+,可学习的旋转向量,以及用于错误传播和重建的双分支优化.
主要成果:
- LRQuant和LRQuant+在各种LFM中表现出卓越的性能,包括LLM,ViTS和MLLM.
- 这些方法在重量激活和仅重量量化方面都取得了有效性,特别是在挑战W4A4和W2A16场景时.
- 实验结果验证了拟议的LRQuant框架的统一适用性和稳定性.
结论:
- LRQuant在LFM的培训后量化方面取得了重大进展,提高了效率和准确性.
- 可学习的方法和新的NLC损失有效地减轻了量子化错误,并提高了模型的稳定性.
- 该框架在不同模型和量子化比特宽度的多功能性使其成为部署LFM的有价值工具.
相关概念视频
Improving Translational Accuracy
11.9K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
11.9K
Estimation of the Physical Quantities
5.9K
On many occasions, physicists, other scientists, and engineers need to make estimates of a particular quantity. These are sometimes referred to as guesstimates, order-of-magnitude approximations, back-of-the-envelope calculations, or Fermi calculations. The physicist Enrico Fermi was famous for his ability to estimate various kinds of data with surprising precision. Estimating does not mean guessing a number or a formula at random. Instead, estimation means using prior experience and sound...
5.9K
Transformers with Off-Nominal Turns Ratios
205
In scenarios involving parallel transformers with disparate ratings, developing per-unit models requires accommodating off-nominal turns ratios. This situation arises when the selected base voltages are not proportional to the transformer’s voltage ratings. Consider a transformer where the rated voltages are related by the term a. If the chosen voltage bases satisfy a relationship involving term b, term c is defined as the ratio of these bases. This ratio is then substituted into the...
205
The Ideal Transformer
890
In single-phase two-winding transformers, two windings are coiled around a magnetic core characterized by cross-sectional area A and magnetic permeability μ. A phasor current i1 enters the left winding while i2 exits the right winding, establishing the fundamental working of the transformer through electromagnetic principles.
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
890
Energy Losses in Transformers
974
In an ideal transformer, it is assumed that there are no energy losses, and, hence, all the power at the primary winding is transferred to the secondary winding. However, in reality, the transformers always have some energy losses, and, hence, the output power obtained at the secondary winding is less than the input power at the primary winding due to energy losses.
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
There are four main reasons for energy losses in transformers.
The first cause can be the high resistance of the...
974
Linear Approximation in Frequency Domain
131
Linear systems are characterized by two main properties: superposition and homogeneity. Superposition allows the response to multiple inputs to be the sum of the responses to each individual input. Homogeneity ensures that scaling an input by a scalar results in the response being scaled by the same scalar.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
131


