QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Post-training quantization (PTQ) is crucial for reducing the memory footprint of large language models (LLMs).
• Existing PTQ methods face challenges in extreme compression regimes (e.g., ≤ 4 bits per weight).

Purpose of the Study:

• Introduce QuIP#, a novel weight-only PTQ method for extreme LLM compression.
• Achieve state-of-the-art results in extreme quantization by employing three innovative techniques.

Main Methods:

• Utilize the randomized Hadamard transform for improved incoherence processing.
• Employ vector quantization with hardware-efficient codebooks based on the E8 lattice for sub-Gaussian weights.
• Incorporate fine-tuning to enhance fidelity to the original model.

Main Results:

• QuIP# demonstrates superior performance compared to existing PTQ methods in extreme compression.
• The method enables new scaling behaviors for PTQ.
• QuIP# supports fast inference speeds.

Conclusions:

• QuIP# represents a significant advancement in weight-only PTQ for LLMs.
• The novel techniques employed lead to state-of-the-art results in extreme compression.
• QuIP# offers a practical solution for deploying highly compressed LLMs efficiently.