Stabilizing iterative pruning with local adjustments and global scaling learning rate.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Iterative pruning reduces model complexity but causes weight importance instability.
• Structural imbalance from zeroed/masked weights leads to fluctuating importance scores, hindering pruning decisions and retraining.

Purpose of the Study:

• To introduce LAGA (Local Adjustments and Global Scaling), a dynamic learning rate adaptation strategy.
• To stabilize importance estimation during iterative pruning and improve pruned model performance.

Main Methods:

• LAGA dynamically adjusts learning rates based on pruning status: local adjustments smooth temporal signals and compensate reactivation, while global scaling adapts to model sparsity.
• The strategy rebalances training dynamics to recover stable importance trajectories.

Main Results:

• LAGA significantly improves importance evaluation stability and pruned model performance.
• Pruning ViT-B/16 by 60% on CIFAR-100 with LAGA yielded 4.71% higher Top-1 accuracy than AdamW.
• Pruning DeiT-S by 40% on CIFAR-100 showed a 0.68% Top-1 improvement over the unpruned baseline.

Conclusions:

• LAGA demonstrates robustness and generalizability across ViT-B/16, DeiT-S, and Swin-B models on CIFAR-100 and ImageNet-1K datasets.
• The proposed method effectively stabilizes iterative pruning and enhances overall model performance.