PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization.

Area of Science:

• Artificial Intelligence
• Computer Science

Background:

• Existing quantization methods for Large Language Models (LLMs) often overlook token-wise outliers, hindering model accuracy.
• Channel-wise outlier handling is common, but token-wise outliers remain a significant challenge for quantized LLMs.

Purpose of the Study:

• To introduce PrefixQuant, a novel quantization method designed to address token-wise outliers in LLMs.
• To achieve state-of-the-art performance in LLM quantization across different precision levels and granularities.

Main Methods:

• PrefixQuant isolates token-wise outliers by prefixing them in the KV cache, a training-free and efficient process.
• The method incorporates new trainable parameters for block-wise training to mitigate quantization errors.

Main Results:

• PrefixQuant significantly outperforms existing dynamic quantization methods, showing substantial accuracy improvements on zero-shot reasoning tasks.
• Achieved average accuracy gains of +3.08 and +2.85 points over SpinQuant under dynamic and static quantization settings, respectively (W4A4KV4 Llama-3-8B).
• Demonstrated significant speedups in prefilling (up to 2.74×) and decoding (up to 2.16×) using W4A4 PrefixQuant.

Conclusions:

• PrefixQuant offers a highly effective solution for LLM quantization by tackling token-wise outliers.
• The method achieves superior accuracy and efficiency, making it a valuable advancement for deploying LLMs.
• PrefixQuant enables faster and more accurate LLM inference through optimized quantization strategies.