REAF: Remembering Enhancement and Entropy-Based Asymptotic Forgetting for Filter Pruning.

Area of Science:

• Deep learning
• Computer vision
• Model compression

Background:

• Filter pruning is crucial for efficient deep learning models.
• Existing methods often cause irreversible information loss and suboptimal performance due to premature forgetting.
• The baseline model's performance can limit the potential of pruned networks.

Purpose of the Study:

• To develop a novel filter pruning paradigm, REAF, that enhances memory retention and employs asymptotic forgetting.
• To overcome the limitations of current filter pruning techniques by preventing unrecoverable information loss.
• To improve the performance ceiling of slimmed deep learning models.

Main Methods:

• Introduced Remembering Enhancement and Entropy-based Asymptotic Forgetting (REAF) paradigm.
• Utilized fusible compensatory convolutions to enhance baseline remembering without inference cost.
• Implemented a bilateral-collaborated pruning criterion based on filter distance and memory enhancement.
• Applied Ebbinghaus curve-based asymptotic forgetting to stabilize learning during pruning.

Main Results:

• REAF significantly outperforms existing state-of-the-art (SOTA) filter pruning methods.
• Demonstrated substantial reduction in FLOPs (47.55%) and parameters (42.98%) for ResNet-50 on ImageNet.
• Achieved minimal Top-1 accuracy loss (0.98%) on ImageNet, showcasing high efficiency and effectiveness.
• The proposed method liberates pruned models from baseline limitations and prevents catastrophic forgetting.

Conclusions:

• REAF offers a superior approach to filter pruning by balancing memory enhancement and controlled forgetting.
• The method enables significant model compression while maintaining high accuracy, crucial for deploying deep learning models.
• REAF provides a robust and effective strategy for optimizing deep neural networks for efficiency.