Dual-Grained Lightweight Strategy.

Area of Science:

• Deep learning model optimization
• Artificial intelligence for edge computing
• Computational efficiency in neural networks

Background:

• Model compression techniques like pruning at initialization are crucial for deploying deep neural networks on resource-constrained edge devices.
• Current pruning methods struggle with performance degradation at extreme sparsity levels.
• Simplifying deep neural networks reduces storage, speeds up training and inference, and lowers energy consumption.

Purpose of the Study:

• To improve the performance of deep neural network models under extreme sparsity conditions.
• To introduce a novel dual-grained lightweight strategy named TEDEPR.
• To leverage tensor theory for optimizing sparse sub-network structures before model training.

Main Methods:

• TEDEPR employs a dual-grained approach: coarse-grained low-rank tensor decomposition and fine-grained weight pruning.
• Coarse-grained level: Weight matrices/tensors are decomposed into low-rank forms using multi-step chain operations to enhance feature extraction.
• Fine-grained level: Unimportant weights are pruned based on trainability in the low-rank model prior to training.

Main Results:

• Extensive experiments on diverse datasets (MNIST, CIFAR, ImageNet) and architectures (LeNet, ResNet, Transformer) demonstrate TEDEPR's effectiveness.
• TEDEPR achieves higher accuracy compared to existing pruning at initialization methods under extreme sparsity.
• The method results in faster training and inference times and reduced model storage space.

Conclusions:

• TEDEPR offers a superior approach to pruning at initialization, particularly for achieving high performance at extreme sparsity.
• The integration of tensor theory provides a novel mechanism for optimizing sparse sub-network structures.
• TEDEPR facilitates the deployment of high-performance models on edge devices, advancing intelligent systems.