LRPRNet: Lightweight Deep Network by Low-Rank Pointwise Residual Convolution

Deep learning has become popular in recent years primarily due to powerful computing devices such as graphics processing units (GPUs). However, it is challenging to deploy these deep models to multimedia devices, smartphones, or embedded systems with limited resources. To reduce the computation and memory costs, we propose a novel lightweight deep learning module by low-rank pointwise residual (LRPR) convolution, called LRPRNet. Essentially, LRPR aims at using a low-rank approximation in pointwise convolution to further reduce the module size while keeping depthwise convolutions as the residual module to rectify the LRPR module. This is critical when the low-rankness undermines the convolution process. Moreover, our LRPR is quite general and can be directly applied to many existing network architectures such as MobileNetv1, ShuffleNetv2, MixNet, and so on. Experiments on visual recognition tasks, including image classification and face alignment on popular benchmarks, show that our LRPRNet achieves competitive performance but with a significant reduction of Flops and memory cost compared to the state-of-the-art deep lightweight models.