FRM-PTQ: Feature relationship matching enhanced low-bit post-training quantization for large language models.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Post-Training Quantization (PTQ) is crucial for reducing memory and computation in Large Language Models (LLMs) inference.
• Existing PTQ methods struggle with ultra-low-bit quantization, leading to performance degradation, especially in advanced models like LLaMA-3 and LLaMA-3.1.

Purpose of the Study:

• To develop a novel PTQ framework, FRM-PTQ, that mitigates performance loss in ultra-low-bit quantization.
• To enhance the efficiency and accuracy of LLMs through advanced quantization techniques.

Main Methods:

• Introduced Feature Relationship Matching (FRM) to model token-level relationships and structure-level distributions via intra-block self-distillation.
• Employed feature relationship matching in high-dimensional spaces to bridge the gap between quantized and full-precision blocks, moving beyond conventional MSE loss.
• Developed a multi-granularity per-group quantization technique with a customized kernel based on decoder block sensitivity.

Main Results:

• FRM-PTQ achieves outstanding performance in W4A4 low-bit quantization, maintaining near full-precision accuracy.
• Demonstrated a 2x throughput improvement and 3.17x memory reduction.
• Showcased significant advantages in latest models (LLaMA-3, LLaMA-3.1, Qwen2.5) and extreme low-bit scenarios (W3A3).

Conclusions:

• FRM-PTQ effectively addresses the performance degradation issues in ultra-low-bit quantization for LLMs.
• The proposed feature relationship matching and multi-granularity quantization offer a robust solution for efficient LLM deployment.
• FRM-PTQ provides a practical and effective method for deploying state-of-the-art LLMs with reduced resource requirements.