Filter Sketch for Network Pruning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) are computationally intensive, requiring significant resources for training and deployment.
• Network pruning is a crucial technique for reducing the size and complexity of DNNs, but often involves substantial time costs and potential accuracy degradation.

Purpose of the Study:

• To introduce a novel and efficient network pruning approach that preserves essential information from pretrained network weights.
• To significantly reduce the computational cost and parameter count of deep neural networks.

Main Methods:

• The proposed FilterSketch method formulates network pruning as a matrix sketch problem.
• It utilizes the off-the-shelf frequent direction method to efficiently solve the matrix sketch problem.
• FilterSketch encodes the second-order information of pretrained weights, enabling representation capacity recovery through simple fine-tuning.

Main Results:

• FilterSketch achieved a 63.3% reduction in floating-point operations (FLOPs) and pruned 59.9% of parameters on ResNet-110 for CIFAR-10 with negligible accuracy loss.
• On ILSVRC-2012, FilterSketch reduced FLOPs by 45.5% and parameters by 43.0% for ResNet-50, with only a 0.69% accuracy drop.
• The method demonstrated a significant reduction in time cost for pruning optimization, requiring neither training from scratch nor iterative optimization.

Conclusions:

• FilterSketch offers an efficient and effective solution for deep neural network pruning.
• The approach successfully reduces model complexity while maintaining high accuracy, making DNNs more accessible for deployment.
• Its speed and minimal accuracy impact make it a valuable tool for optimizing large-scale neural networks.