Efficient Neural Network Compression Inspired by Compressive Sensing.

Area of Science:

• Artificial Intelligence
• Signal Processing
• Machine Learning

Background:

• Traditional neural network compression (NNC) methods reduce model size and FLOPs by removing unimportant weights.
• Existing NNC methods have not fully exploited the intrinsic sparsity characteristics of neural networks.

Purpose of the Study:

• To propose a compressive sensing (CS)-based method, NNCS, for improved neural network compression.
• To exploit greater sparsity of weight parameters in the transform domain compared to the original domain.

Main Methods:

• Incorporated a constrained CS model into the loss function to achieve sparse representations in the transform domain during training.
• Developed a two-step training process: training raw weights and their sparse representations, then training transform coefficients.
• Transformed the neural network into a new domain representation for sparser parameter distribution and inference acceleration.

Main Results:

• NNCS significantly outperforms state-of-the-art methods in parameter reduction and FLOPs reduction.
• Achieved 94.8% parameter and 76.8% FLOPs reduction on VGGNet (CIFAR-10) with minimal accuracy drop (0.13%).
• Achieved 75.6% parameter and 78.9% FLOPs reduction on ResNet-50 (ImageNet) with a 1.24% accuracy drop.

Conclusions:

• NNCS effectively enhances neural network compression by utilizing transform-domain sparsity.
• The proposed method offers significant improvements in model size and computational efficiency.
• NNCS demonstrates superior performance over existing compression techniques for deep learning models.