STPP: Efficient and Progressive Structured Pruning Via Enhanced Sparsification Paradigm

As an effective technique for network compression, sparsification-based pruning generally applies penalty terms to suppress the importance of dropped parameters, which is regarded as the suppressed sparsification paradigm. This paradigm weakens the dropped parameters, damaging the capacity of the dense network before pruning and thereby leading to performance degradation. To address the above issue, this paper comprehensively investigates the relative sparsity effect of emerging stimulative training (ST) and reveals the potential of the enhanced sparsification to alleviate performance degradation. Based on the relative sparsity effect, a structured pruning framework named STPP is proposed, utilizing an enhanced sparsification paradigm rather than a conventional suppressed one. STPP maintains the magnitude of dropped parameters and enhances the expressivity of kept parameters by self-distillation. To mitigate the suboptimality of subnet architecture exploration and severe distillation gap in vanilla ST, three key designs are introduced in STPP: (1) Efficient KD-guided exploration with architecture-aware metric broadcast. STPP couples enhanced sparsification and architecture exploration into a unified process guided by KD loss, and utilizes architecture-aware metric broadcast for efficient architecture exploration. (2) Progressive sparsification suitable for multi-dimensional space. STPP gradually reduces the width of subnets to transfer the expressivity smoothly and alleviate the distillation gap. (3) Subnet mutating expansion. STPP introduces a support subnet mutated from the sampled subnet for hierarchical knowledge distillation. Based on the above techniques, the potential of the novel enhanced sparsification paradigm in structured pruning is sufficiently unleashed. Specifically, without fine-tuning or other bells and whistles, STPP can reach a new Pareto frontier at different budgets compared to existing pruning methods, especially under extremely aggressive pruning scenarios, e.g., remaining 97.32% Top-1 accuracy (74.11% in 76.15%) while reducing 85% FLOPs for ResNet-50 on ImageNet.