Efficient Distortion-Minimized Layerwise Pruning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep Neural Networks (DNNs) are computationally intensive, requiring efficient methods for model compression.
• Pruning, a technique to reduce model size and complexity, often faces challenges in optimizing layer-wise reductions and minimizing output distortion.

Purpose of the Study:

• To develop a post-training pruning framework that jointly optimizes layer-wise pruning to minimize model output distortion.
• To leverage a newly discovered additivity property of output distortion for efficient pruning optimization.

Main Methods:

• A post-training pruning framework is proposed, optimizing layer-wise pruning to minimize model output distortion.
• An additivity property of output distortion in DNNs is identified and utilized.
• Pruning optimization is reformulated as a combinatorial problem solved with dynamic programming, achieving linear time complexity.
• Hessian-based Taylor approximation is employed to optimize distortions, enhancing pruning efficiency.

Main Results:

• The framework achieves state-of-the-art (SoTA) results with significant FLOPs reductions across various DNN architectures (CNNs, ViTs) and tasks (image classification, 3D object detection).
• Examples include up to 29.2x FLOPs reduction on CIFAR-10 (VGG-16) and no accuracy loss with 2x FLOPs reduction on ImageNet (DeiT-Base).
• Significant FLOPs reductions were also observed in 3D object detection models like CenterPoint (3.89x) and PVRCNN (3.72x).

Conclusions:

• The proposed layer-adaptive weight pruning framework is effective and practical for enhancing model performance.
• The method demonstrates significant computational savings (FLOPs reduction) without accuracy degradation.
• The dynamic programming approach based on the additivity property offers a fast and efficient solution for DNN pruning.