SeReNe: Sensitivity-Based Regularization of Neurons for Structured Sparsity in Neural Networks

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) possess millions of parameters, hindering deployment on devices with limited resources.
• Efficient model compression is crucial for enabling DNNs on edge devices.

Purpose of the Study:

• To introduce Sensitivity-based regularization of neurons (SeReNe) for structured sparsity in DNNs.
• To enable practical network footprint reduction by pruning entire neurons.

Main Methods:

• Defined neuron sensitivity as the network output's variation concerning neuron activity variation.
• Incorporated neuron sensitivity into the cost function as a regularization term.
• Pruned neurons exhibiting low sensitivity.

Main Results:

• Demonstrated practical network footprint reduction by pruning whole neurons.
• Achieved competitive compression ratios across various network architectures and datasets.
• Showcased the effectiveness of SeReNe compared to state-of-the-art methods.

Conclusions:

• SeReNe effectively reduces DNN size by pruning insensitive neurons.
• The method facilitates the deployment of DNNs on resource-constrained hardware.
• Sensitivity-based regularization offers a promising approach for efficient neural network compression.