Co-Exploring Structured Sparsification and Low-Rank Tensor Decomposition for Compact DNNs.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural network (DNN) models are computationally intensive.
• Model compression techniques like sparsification and low-rank decomposition are crucial for efficient deployment.
• Integrating these techniques, especially structured variants, remains underexplored.

Purpose of the Study:

• To systematically co-explore structured sparsification and tensor decomposition for compact DNN models.
• To analyze key design factors influencing their joint application.
• To propose an efficient, unified framework for integrated compression.

Main Methods:

• Investigated design factors: operational sequence, decomposition format, and optimization procedure.
• Developed CEPD, a unified framework for co-exploring structured sparsification and tensor decomposition.
• Conducted empirical experiments on CIFAR-10 and ImageNet datasets.

Main Results:

• CEPD achieved accuracy increases of 0.72%-0.45% on CIFAR-10 (ResNet-56, MobileNetV2) with 43.0%-44.2% computational cost reduction.
• On ImageNet, CEPD improved accuracy by 0.10%-1.39% (ResNet-18, ResNet-50) while reducing parameters by 59.4%-54.6%.

Conclusions:

• The proposed CEPD framework effectively integrates structured sparsification and tensor decomposition.
• This co-exploration yields significant improvements in DNN model compression, balancing accuracy and efficiency.
• CEPD demonstrates a promising approach for developing more compact and performant deep learning models.