A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Modern deep neural networks (DNNs), especially large language models (LLMs), demand substantial computational and storage resources.
• Model compression techniques like pruning are crucial for deploying DNNs in resource-constrained environments and accelerating inference.
• Despite over 3,000 pruning papers published recently, a comprehensive, up-to-date review is lacking.

Purpose of the Study:

• To provide a comprehensive review of deep neural network pruning research.
• To establish a taxonomy for understanding pruning techniques.
• To identify emerging trends and future research directions in model compression.

Main Methods:

• Categorization of pruning research into four main areas: speedup type, pruning timing, pruning methodology, and integration with other compression techniques.
• Comparative analysis of eight key pruning strategy pairs (e.g., unstructured vs. structured, one-shot vs. iterative).
• Exploration of pruning applications in LLMs, vision transformers, diffusion models, and multimodal models, alongside post-training and supervised pruning methods.

Main Results:

• A structured overview of existing deep neural network pruning research.
• Comparative insights into different pruning strategies and their trade-offs.
• Identification of emerging topics and their specific challenges and opportunities.

Conclusions:

• The survey provides a foundation for developing new pruning methods by highlighting commonalities and differences.
• Recommendations are offered for selecting appropriate pruning techniques.
• Promising future research avenues in neural network pruning are outlined, including applications in adversarial robustness and natural language understanding.