QuIP:具有保证的大型语言模型的2位量子化.
Jerry Chee1, Yaohui Cai1, Volodymyr Kuleshov1
1Cornell University.
Advances in neural information processing systems
|October 17, 2024
概括
量子化与不连贯处理 (QuIP) 通过使权重和赫森矩阵不连贯来增强大型语言模型 (LLM). 这种方法使每重量仅使用两位的可行LLM量子化.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 计算机科学 计算机科学
背景情况:
- 培训后的参数量化对于有效地部署大型语言模型 (LLM) 是至关重要的.
- 现有的量子化方法在保持非常低位精度的模型性能方面面临挑战.
研究的目的:
- 引入一种新的量子化方法,即用不一致处理进行量子化 (QuIP),用于LLMs.
- 从理论上分析LLM级量的定量化算法,并证明重量和赫斯不一致性的好处.
主要方法:
- 在QuIP中,采用了自适应圆形化程序,以最大限度地减少二次代理目标.
- 它使用随机直角矩阵的预处理和后处理来诱导权重和黑斯不连贯性.
- 提供了QuIP的理论分析,并扩展到OPTQ算法.
主要成果:
- 不相干预处理可以明显改进现有的量子化算法.
- QuIP实现了第一个可行的LLM量子化,使用每重量只有两位.
- 开发的理论框架适用于其他量子化方法,如OPTQ.
结论:
- 权重和赫森不一致性是LLM定量化的一个有益的属性.
- 在有效的LLM部署中,QuIP提供了显著的进步.
- 提出的方法和理论分析为进一步研究低位LLM量子化铺平了道路.
更多相关视频
相关概念视频
Improving Translational Accuracy
9.3K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
9.3K
Per-Unit Sequence Models
72
An ideal Y-Y transformer, grounded through neutral impedances, displays per-unit sequence networks akin to those of a single-phase ideal transformer when subjected to balanced positive- or negative-sequence currents. These currents do not produce neutral currents, and their associated voltage drops.
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
Zero-sequence currents, which are identical in magnitude and phase, generate a neutral current, resulting in voltage drops across the neutral impedance and the low-voltage winding. If the...
72
Accuracy, limits, and approximation
441
Accuracy, limits, and approximations are common in many fields, especially in engineering calculations. These concepts are imperative for ensuring that a given value is as close as possible to its true value.
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
441
Propagation of Uncertainty from Random Error
654
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
654
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
44
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
44
Interpretation of Confidence Intervals
5.6K
A confidence interval is a better estimate of the population than a point estimate, as it uses a range of values from a sample instead of a single value.
Confidence intervals have confidence coefficients that are crucial for their interpretation. The most common confidence coefficients are 0.90, 0.95, and 0.99, which can be written as percentages–90%, 95%, and 99%, respectively.
Suppose a person calculates a confidence interval with a confidence coefficient of 0.95. In that case, they can...
Confidence intervals have confidence coefficients that are crucial for their interpretation. The most common confidence coefficients are 0.90, 0.95, and 0.99, which can be written as percentages–90%, 95%, and 99%, respectively.
Suppose a person calculates a confidence interval with a confidence coefficient of 0.95. In that case, they can...
5.6K


