Implementation of Lightweight Convolutional Neural Networks via Layer-Wise Differentiable Compression.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Convolutional Neural Networks (CNNs) excel in various fields but suffer from large sizes and high computational demands, limiting their use on devices with limited resources.
• Efficient compression of CNNs is crucial to mitigate size and computational costs without sacrificing performance.

Purpose of the Study:

• To propose a novel layer-wise differentiable compression (LWDC) algorithm for structural compression of CNNs.
• To enable simultaneous compression and training of CNNs using gradient descent.
• To offer a generic compressed model requiring no specialized hardware or software.

Main Methods:

• Introduced a differentiable selection operator (OS) embedded within the CNN architecture.
• Implemented a method that directly replaces bulky operators with lightweight alternatives during training.
• Utilized gradient descent for simultaneous compression and training, optimizing for specified lightweight operators and a regularization factor.

Main Results:

• LWDC achieved significant compression rates on CIFAR-10, CIFAR-100, and ImageNet datasets.
• The method demonstrated superior compression compared to state-of-the-art techniques in most cases.
• LWDC resulted in lower performance degradation than existing compression methods.

Conclusions:

• The proposed LWDC algorithm effectively compresses CNNs structurally, reducing model size and computational cost.
• LWDC offers a viable solution for deploying deep learning models on resource-constrained devices.
• The study highlights the effectiveness of replacing operators directly and evaluates the impact of operator choice and regularization.