Reweighted Alternating Direction Method of Multipliers for DNN weight pruning

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep Neural Networks (DNNs) face computational challenges due to increasing complexity and size.
• Weight pruning is a key technique for optimizing DNNs by reducing model size and computational cost.
• Existing pruning methods often lack efficiency or require extensive hyperparameter tuning.

Purpose of the Study:

• To propose a novel dynamic regularization-based pruning method for DNNs.
• To integrate the Alternating Direction Method of Multipliers (ADMM) with a reweighting mechanism for improved weight importance assignment.
• To reduce computational burden and hyperparameter dependency in DNN optimization.

Main Methods:

• Developed a dynamic regularization pruning approach incorporating ADMM.
• Introduced a reweighting mechanism to dynamically assign importance to network weights.
• Evaluated the method on various DNN architectures (LeNet-5, ResNet-32, ResNet-56, ResNet-50) and datasets (MNIST, CIFAR-10, ImageNet).

Main Results:

• Achieved superior compression ratios and accuracy compared to state-of-the-art pruning methods.
• Demonstrated significant compression (355.9×) on LeNet-5 (MNIST) with accuracy improvement.
• Obtained substantial compression (4.24×) on ResNet-50 (ImageNet) without accuracy loss.
• Showcased reduced hyperparameter requirements, saving considerable time.

Conclusions:

• The proposed ADMM-based dynamic regularization pruning method effectively optimizes DNNs.
• The reweighting mechanism enhances weight importance assignment, leading to better performance.
• The method offers a significant advancement in efficient and accurate DNN compression.