Enabling efficient low-bit quantization based on matrix product operators for KV cache compression.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Large language models (LLMs) are computationally expensive to deploy.
• Key-value (KV) cache is vital for LLM inference speed but demands significant memory.
• Existing KV cache compression methods often compromise accuracy or require calibration data.

Purpose of the Study:

• To introduce MPOQ, a novel data-free quantization technique for compressing LLM KV caches.
• To reduce the memory footprint of LLMs without degrading performance.
• To offer a practical solution for real-world LLM applications.

Main Methods:

• Developed MPOQ, a quantization technique utilizing matrix product operators (MPO).
• MPO decomposes matrices into local tensors, enabling targeted quantization.
• Employs a hybrid quantization strategy: low-bit for large tensors, high-precision for smaller tensors containing outliers.

Main Results:

• Achieved approximately 75% reduction in KV cache memory footprint.
• Maintained comparable generation quality to uncompressed models.
• Demonstrated effectiveness across various LLMs including OPT, LLaMA, and Mistral.

Conclusions:

• MPOQ offers an effective data-free approach to compress LLM KV caches.
• The method significantly enhances LLM efficiency and reduces memory costs.
• MPOQ presents a viable strategy for practical LLM deployment.