An effective low-rank compression with a joint rank selection followed by a compression-friendly training

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Low-rank compression is a key technique for reducing neural network size.
• Existing methods face challenges in optimal rank determination and compression-friendly training.

Purpose of the Study:

• To propose BSR (Beam-search and Stable Rank), an algorithm addressing challenges in neural network low-rank compression.
• To introduce an efficient rank-selection method and a unique compression-friendly training approach.

Main Methods:

• BSR utilizes a modified beam search for joint optimization of rank allocations across all layers.
• A regularization loss derived from a modified stable rank is employed for rank control with minimal performance impact.

Main Results:

• BSR demonstrates superior performance compared to existing low-rank compression methods.
• On CIFAR10 (ResNet56), BSR achieved compression with performance improvement up to a 0.82 compression ratio.
• BSR outperformed the state-of-the-art LC method on CIFAR100 (ResNet56) and ImageNet (AlexNet) by 4.7% and 6.7% on average, respectively.

Conclusions:

• BSR is an effective low-rank compression algorithm that overcomes key challenges in the field.
• The method shows broad applicability, performing well on various architectures including EfficientNet-B0 and MobileNetV2.
• BSR offers competitive performance against pruning methods and can be combined with quantization for further compression.