R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Reasoning models excel at complex tasks but generate lengthy outputs, increasing inference costs due to large Key-Value (KV) caches.
• Existing KV cache compression methods struggle with chain-of-thought reasoning, leading to performance degradation and reasoning failures.

Purpose of the Study:

• To develop a novel KV cache compression technique tailored for reasoning models to mitigate excessive memory usage.
• To improve the efficiency of chain-of-thought reasoning without compromising performance.

Main Methods:

• Propose Redundancy-aware KV Cache Compression for Reasoning models (R-KV), a method focusing on compressing redundant tokens within reasoning processes.
• Evaluate R-KV's effectiveness against existing KV cache compression baselines on mathematical reasoning tasks.

Main Results:

• R-KV preserves nearly 100% of full KV cache performance using only 10% of the cache, significantly outperforming baselines (60% performance).
• Achieves 105% of full KV cache performance with 16% cache, demonstrating superior efficiency.
• Reduces memory by 90% and increases throughput by 6.6x compared to standard chain-of-thought inference.

Conclusions:

• R-KV effectively compresses KV caches for reasoning models, addressing the challenge of long outputs and high memory consumption.
• The proposed method offers substantial memory savings and throughput improvements while maintaining or enhancing performance on complex reasoning tasks.