Feature flow regularization: Improving structured sparsity in deep neural networks.

Area of Science:

• Deep Learning
• Computer Vision
• Artificial Intelligence

Background:

• Deep neural networks (DNNs) require compression for efficient deployment.
• Existing pruning methods often impose direct constraints on network parameters.
• A novel approach is needed to improve structured pruning and sparsity in DNNs.

Purpose of the Study:

• To propose a new regularization strategy for structured pruning in DNNs.
• To leverage the concept of feature evolution, termed feature flow, for model compression.
• To enhance structured sparsity and network efficiency through regularization.

Main Methods:

• Introduced Feature Flow Regularization (FFR) as a novel regularization strategy.
• FFR penalizes the length and total absolute curvature of feature trajectories between adjacent layers.
• This implicitly promotes structured sparsity in DNN parameters.

Main Results:

• FFR effectively improves structured sparsity in DNNs.
• The proposed method achieves pruning results comparable to or exceeding state-of-the-art techniques.
• Experiments conducted on CIFAR-10 and ImageNet datasets validate the effectiveness of FFR.

Conclusions:

• Feature Flow Regularization offers a simple yet effective method for structured pruning.
• FFR provides a new perspective on improving DNN efficiency by analyzing feature evolution.
• The approach leads to more efficient networks by avoiding redundant parameters.