A survey of model compression techniques: past, present, and future

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• General-purpose large models show exceptional performance, driving domain-specific model development.
• Large models demand substantial computational resources and memory during training and inference.
• Practical deployment of large models faces significant hardware and resource challenges.

Purpose of the Study:

• To provide a comprehensive review of model compression techniques.
• To explore fundamental principles, recent advancements, and innovative strategies in model compression.
• To offer insights into practical applications and future directions for efficient model deployment.

Main Methods:

• Review of model compression evolution.
• Detailed examination of quantization, pruning, low-rank decomposition, and knowledge distillation.
• Analysis of recent advancements and innovative strategies in these methods.

Main Results:

• Model compression is vital for overcoming resource limitations of large models.
• Various techniques like quantization and pruning offer pathways to efficient deployment.
• The review synthesizes current knowledge and identifies future research directions.

Conclusions:

• Model compression is crucial for the practical and widespread deployment of large AI models.
• Understanding diverse compression strategies is key for researchers and practitioners.
• This review provides a foundational resource for advancing efficient AI model development and application.