Towards efficient network compression via Few-Shot Slimming.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Network compression methods typically require extensive labeled training data, which is often impractical due to privacy, storage, or transmission constraints.
• Few-shot learning offers a promising alternative for model compression when data is scarce.

Purpose of the Study:

• To propose a novel few-shot network compression framework, Few-Shot Slimming (FSS), designed to operate effectively with minimal unlabeled data.
• To address the limitations of traditional network compression techniques that heavily rely on large datasets.

Main Methods:

• FSS employs a student/teacher paradigm involving two key steps: inheriting principal feature maps from the teacher and refining the student's representation.
• Feature map selection utilizes normalized cross-correlation and a novel indicator based on feature map variances for information richness.
• Knowledge distillation is enhanced with GridMix, a novel grid-based mixing data augmentation technique, to improve student model performance.

Main Results:

• Experiments demonstrate FSS achieves state-of-the-art performance on multiple benchmarks.
• Using only 0.2% of label-free data, FSS reduced FLOPs by 60% for DenseNet-40 on CIFAR-10 with a minimal 0.8% accuracy loss.
• The results are comparable to conventional methods that use full training datasets.

Conclusions:

• Few-Shot Slimming (FSS) offers an effective solution for network compression in data-limited scenarios.
• The framework successfully leverages unlabeled data for significant model compression while maintaining high accuracy.
• FSS demonstrates the potential of few-shot learning in advancing practical deep learning applications.