An effective low-rank compression with a joint rank selection followed by a compression-friendly training

Low-rank compression of a neural network is one of the popular compression techniques, where it has been known to have two main challenges. The first challenge is determining the optimal rank of all the layers and the second is training the neural network into a compression-friendly form. To overcome the two challenges, we propose BSR (Beam-search and Stable Rank), a low-rank compression algorithm that embodies an efficient rank-selection method and a unique compression-friendly training method. For the rank selection, BSR employs a modified beam search that can perform a joint optimization of the rank allocations over all the layers in contrast to the previously used heuristic methods. For the compression-friendly training, BSR adopts a regularization loss derived from a modified stable rank, which can control the rank while incurring almost no harm in performance. Experiment results confirm that BSR is effective and superior when compared to the existing low-rank compression methods. For CIFAR10 on ResNet56, BSR not only achieves compression but also provides a performance improvement over the baseline model's performance for the compression ratio of up to 0.82. For CIFAR100 on ResNet56 and ImageNet on AlexNet, BSR outperforms the previous SOTA method, LC, by 4.7% and by 6.7% on the average, respectively. BSR is also effective for EfficientNet-B0 and MobileNetV2 that are known for their efficient design in terms of parameters and computational cost. We also show that BSR provides a competitive performance when compared with the recent pruning compression algorithms. As with pruning, BSR can be easily combined with quantization for an additional compression.