Improving Lightweight AdderNet via Distillation From ℓ₂ to ℓ₁-norm

To achieve efficient inference with a hardware-friendly design, Adder Neural Networks (ANNs) are proposed to replace expensive multiplication operations in Convolutional Neural Networks (CNNs) with cheap additions through utilizing l₁ -norm for similarity measurement instead of cosine distance. However, we observe that there exists an increasing gap between CNNs and ANNs with reducing parameters, which cannot be eliminated by existing algorithms. In this paper, we present a simple yet effective Norm-Guided Distillation (NGD) method for l₁ -norm ANNs to learn superior performance from l₂ -norm ANNs. Although CNNs achieve similar accuracy with l₂ -norm ANNs, the clustering performance based on l₂ -distance can be easily learned by l₁ -norm ANNs compared with cross correlation in CNNs. The features in l₂ -norm ANNs are encouraged to achieve intra-class centralization and inter-class decentralization to amplify this advantage. Furthermore, the roughly estimated gradients in vanilla ANNs are modified to a progressive approximation from l₂ -norm to l₁ -norm so that a more accurate optimization can be achieved. Extensive evaluations on several benchmarks demonstrate the effectiveness of NGD on lightweight networks. For example, our method improves ANN by 10.43% with 0.25× GhostNet on CIFAR-100 and 3.1% with 1.0× GhostNet on ImageNet.