Improving Lightweight AdderNet via Distillation From ℓ₂ to ℓ₁-norm

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Convolutional Neural Networks (CNNs) rely on computationally expensive multiplication operations.
• Adder Neural Networks (ANNs) offer a hardware-friendly alternative by replacing multiplications with additions using the l1-norm.
• A performance gap exists between CNNs and ANNs, particularly with reduced parameters, which current methods fail to address.

Purpose of the Study:

• To introduce a novel method, Norm-Guided Distillation (NGD), to improve the performance of l1-norm ANNs.
• To enable l1-norm ANNs to learn effectively from l2-norm ANNs, bridging the performance gap.

Main Methods:

• Proposed Norm-Guided Distillation (NGD) for l1-norm ANNs.
• Leveraged the superior clustering performance of l2-distance in l2-norm ANNs for feature learning.
• Encouraged intra-class centralization and inter-class decentralization in l2-norm ANNs features.
• Modified ANN gradients for progressive approximation from l2-norm to l1-norm for accurate optimization.

Main Results:

• NGD significantly improves the performance of lightweight ANNs.
• Demonstrated effectiveness on benchmarks like CIFAR-100 and ImageNet.
• Achieved a 10.43% improvement with 0.25x GhostNet on CIFAR-100.
• Achieved a 3.1% improvement with 1.0x GhostNet on ImageNet.

Conclusions:

• NGD is an effective method for enhancing l1-norm ANNs, especially for resource-constrained applications.
• The proposed technique successfully bridges the performance gap between l1-norm and l2-norm based networks.
• NGD offers a practical solution for efficient deep learning inference on hardware-friendly designs.