Hyper-Compression: Model Compression via Hyperfunction

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Science

Background:

• The exponential growth in large model size outpaces available computing resources.
• Existing model compression techniques like pruning and quantization have limitations.

Purpose of the Study:

• To introduce a novel model compression technique called hyper-compression.
• To address the gap between large model size and computing resource limitations.

Main Methods:

• Hyper-compression represents model parameters using dynamic systems as hyperfunctions.
• A dynamic system with irrational winding is identified as the hyperfunction.
• Theoretical error bounds are derived, and engineering optimizations are applied.

Main Results:

• Hyper-compression achieves preferable compression ratios without post-hoc retraining.
• It offers affordable inference and short compression times, compressing LLaMA2-7B in one hour.
• Performance degradation is less than 1%, comparable to int4-quantization.

Conclusions:

• Hyper-compression presents a distinct and effective mechanism for large model compression.
• The method is pragmatic, efficient, and achieves high performance with significant size reduction.