A Multiply-And-Max/Min Neuron Paradigm for Aggressively Prunable Deep Neural Networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) are crucial for Internet of Things (IoT) and mobile AI, but require significant resources.
• Existing pruning techniques reduce DNN size by removing interconnections but can impact performance.
• Traditional DNNs use multiply-and-accumulate (MAC) neurons, which are not inherently optimized for aggressive pruning.

Purpose of the Study:

• To propose a novel neuron structure, Multiply-and-Max/Min (MAM), for creating naturally prunable DNN layers.
• To demonstrate that MAM-based DNNs can achieve high sparsity with negligible performance degradation.
• To evaluate the effectiveness of MAM across various DNN architectures and computer vision datasets.

Main Methods:

• Introduced a novel neuron structure based on a Multiply-and-Max/Min (MAM) map-reduce paradigm.
• Applied various pruning techniques (single-shot, training-dependent, iterative) to MAM-based DNN layers.
• Tested MAM on AlexNet, VGG-16, and ViT-B/16 models using CIFAR-10, CIFAR-100, and ImageNet-1K datasets.

Main Results:

• MAM-based DNN layers exhibit greater interconnection sparsity compared to classic MAC-based layers.
• State-of-the-art pruning techniques are compatible with MAM layers with minimal modifications.
• On ViT-B/16 fine-tuned on ImageNet-1K, MAM layers achieved 0.1% interconnections for a 6% accuracy drop, versus 38.2% for MAC layers.

Conclusions:

• The MAM neuron structure enables naturally and aggressively prunable DNNs for low-resource edge devices.
• MAM significantly reduces computational and storage requirements for DNNs without compromising performance.
• This novel approach offers a promising direction for efficient AI deployment in resource-constrained environments.