Towards performance-maximizing neural network pruning via global channel attention.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Network pruning is crucial for deploying large neural networks on resource-limited devices.
• Static pruning removes fixed network channels, limiting redundancy excavation.
• Dynamic pruning offers higher compression but suffers from memory inefficiency.

Purpose of the Study:

• To develop a static network pruning method that addresses the memory inefficiency of dynamic pruning.
• To propose a novel framework that respects data variability while maintaining static pruning efficiency.
• To achieve state-of-the-art performance and compression ratios through an improved static pruning strategy.

Main Methods:

• A channel attention-based learn-to-rank framework is proposed to determine global channel redundancy.
• Sample-wise channel attentions are aligned to establish a consistent global ranking.
• This global ranking enables static pruning across all data samples.

Main Results:

• The proposed GlobalPru method achieves superior performance compared to state-of-the-art static and dynamic pruning techniques.
• Experiments on ImageNet, SVHN, and CIFAR-10/100 datasets validate the effectiveness of GlobalPru.
• GlobalPru demonstrates significant performance gains and higher compression ratios.

Conclusions:

• GlobalPru offers an effective static alternative to dynamic network pruning, balancing efficiency and performance.
• The channel attention-based learn-to-rank framework successfully unifies sample-specific information for global pruning decisions.
• This method enables efficient deployment of large neural networks on resource-constrained devices.