HMC: Hybrid model compression method based on layer sensitivity grouping.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Deep neural networks often exhibit over-parameterization, leading to redundant model weights.
• This redundancy typically manifests as low-rank and sparse properties within the model's weight matrices.
• Incomplete utilization of these characteristics can result in suboptimal model compression, impacting accuracy and compression ratios.

Purpose of the Study:

• To propose a unified framework for deep model compression that effectively utilizes both low-rank and sparsity characteristics.
• To introduce a novel hybrid model compression method based on sensitivity grouping (HMC).
• To integrate existing additive hybrid compression (AHC) and novel non-additive hybrid compression (NaHC) methods into a single framework.

Main Methods:

• Developed a hybrid model compression framework (HMC) combining low-rank tensor decomposition and structured pruning.
• Introduced a non-additive hybrid compression (NaHC) approach that groups network layers based on sensitivity to different compression techniques.
• Unified AHC and NaHC within the HMC framework to optimize the integration of low-rank and sparsity.

Main Results:

• The proposed HMC method demonstrated a superior trade-off between test accuracy and compression ratio compared to single-strategy methods.
• Experiments on ResNet models showed that HMC outperforms existing compression techniques, including AHC.
• Sensitivity grouping in NaHC allows for better integration of low-rank and sparsity properties than AHC.

Conclusions:

• The HMC framework offers an effective approach to deep model compression by synergistically leveraging low-rank and sparsity.
• The sensitivity grouping strategy in NaHC is crucial for achieving better compression performance.
• HMC provides a significant advancement in balancing model compression and accuracy for deep learning models.