WideTopo: Improving foresight neural network pruning through training dynamics preservation and wide topologies...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Neural network pruning is crucial for computational resource efficiency.
• Existing methods like saliency-based and graph theory-based pruning have limitations.
• Saliency scores can lead to narrow sub-networks; graph methods struggle with pre-trained models.

Purpose of the Study:

• To propose WideTopo, a novel pruning method that identifies structurally stable sub-networks.
• To improve post-training performance by preserving training dynamics and exploring wide network topologies.
• To address limitations of existing pruning techniques, especially in transfer learning.

Main Methods:

• WideTopo integrates Neural Tangent Kernel (NTK) theory and Implicit Target Alignment (ITA) to capture sub-network training dynamics.
• Employs a density-aware saliency score decay strategy to retain effective nodes.
• Utilizes a repeated mask restoration strategy to sustain layer width in sub-networks.

Main Results:

• Extensive validations on CNN and ViT models for image classification and semantic segmentation.
• Demonstrated effectiveness across diverse architectures and model densities, with both random and pre-trained initializations.
• WideTopo achieved competitive post-training performance compared to existing baselines.

Conclusions:

• WideTopo effectively identifies structurally stable sub-networks with improved performance.
• The method is applicable to various architectures and initialization settings, outperforming current baselines.
• Preserving training dynamics and maintaining network width are key to effective neural network pruning.