Non-Structured DNN Weight Pruning-Is It Beneficial in Any Platform?

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Large deep neural networks (DNNs) face energy efficiency challenges due to high DRAM access energy consumption.
• Model compression techniques, including weight pruning and quantization, are crucial for reducing energy usage.
• Quantization is a standard practice for hardware implementations, making pruning effectiveness evaluation on top of quantization essential.

Purpose of the Study:

• To definitively answer whether non-structured or structured pruning is more beneficial when applied with quantization.
• To provide a fair and fundamental comparison methodology for evaluating pruning techniques.
• To guide future research and development in DNN inference acceleration.

Main Methods:

• Developed ADMM-NN-S, an enhanced framework supporting structured pruning, dynamic regulation, and masked retraining.
• Established a rigorous methodology for comparing non-structured and structured pruning based on storage and computation efficiency.
• Evaluated the proposed methods on LeNet-5, AlexNet, and ResNet-50 models.

Main Results:

• ADMM-NN-S achieved significant weight pruning (348× on LeNet-5, 36× on AlexNet, 8× on ResNet-50) with minimal accuracy loss.
• Demonstrated that fully binarized DNNs can maintain lossless accuracy in many cases.
• Structured pruning consistently outperformed non-structured pruning in both storage and computation efficiency under identical accuracy and quantization levels.

Conclusions:

• Structured pruning offers greater potential for DNN inference acceleration compared to non-structured pruning.
• The study provides strong evidence and a baseline for focusing research on structured sparsity in DNNs.
• Encourages the research community to prioritize structured pruning for energy-efficient DNN acceleration.