STPP: Efficient and Progressive Structured Pruning Via Enhanced Sparsification Paradigm.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Sparsification-based pruning, a network compression technique, typically suppresses parameters, degrading dense network capacity and leading to performance loss.
• Existing methods suffer from performance degradation due to the suppressed sparsification paradigm, which weakens parameters before pruning.

Purpose of the Study:

• To investigate the relative sparsity effect of stimulative training (ST) and reveal the potential of enhanced sparsification for alleviating performance degradation.
• To propose STPP, a structured pruning framework employing an enhanced sparsification paradigm for improved network compression.

Main Methods:

• STPP utilizes an enhanced sparsification paradigm, maintaining dropped parameter magnitudes and improving kept parameter expressivity via self-distillation.
• Key designs include KD-guided exploration with architecture-aware metric broadcast, progressive sparsification for multi-dimensional space, and subnet mutating expansion for hierarchical knowledge distillation.

Main Results:

• STPP reaches a new Pareto frontier at different computational budgets compared to existing pruning methods, especially under aggressive pruning scenarios.
• Achieved 97.32% Top-1 accuracy while reducing 85% FLOPs for ResNet-50 on ImageNet, demonstrating significant efficiency gains without fine-tuning.

Conclusions:

• The enhanced sparsification paradigm in STPP effectively alleviates performance degradation associated with traditional pruning methods.
• STPP offers a powerful approach to structured pruning, achieving state-of-the-art performance and efficiency, particularly in resource-constrained environments.