Rethinking Weight Decay for Efficient Neural Network Pruning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Neural network pruning is crucial for model compression and generalization.
• Existing pruning methods face challenges in performance and scalability.
• Weight decay has been historically used to induce sparsity in neural networks.

Purpose of the Study:

• To introduce Selective Weight Decay (SWD) for efficient, continuous pruning.
• To address limitations of current pruning techniques in deep learning.
• To provide a versatile pruning approach applicable to diverse networks and tasks.

Main Methods:

• Developed Selective Weight Decay (SWD), inspired by early weight decay techniques.
• Theoretically grounded SWD using Lagrangian smoothing for continuous pruning.
• Applied SWD across various deep learning tasks, network types, and pruning structures.

Main Results:

• SWD demonstrates favorable performance-to-parameters ratios compared to state-of-the-art methods.
• Effective pruning achieved on challenging datasets including CIFAR-10, Cora, and ImageNet ILSVRC2012.
• SWD offers a scalable and efficient solution for deep neural network compression.

Conclusions:

• Selective Weight Decay (SWD) presents a significant advancement in neural network pruning.
• The approach offers a versatile and effective method for model compression.
• SWD enhances the performance-to-parameter efficiency of deep learning models.