Transformed ℓ₁ regularization for learning sparse deep neural networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Deep Neural Networks (DNNs) are powerful but computationally intensive.
• Large numbers of parameters cause high memory and computation costs.
• DNNs are prone to overfitting with insufficient data, limiting their use on constrained devices.

Purpose of the Study:

• To develop a method for reducing DNN size and computational cost.
• To address challenges of memory, computation, and overfitting in DNNs.
• To enable DNN application on resource-constrained platforms.

Main Methods:

• Introduced a novel non-convex integrated transformed ℓ₁ regularizer for DNNs.
• Applied the regularizer to the weight matrix space for connection-level sparsity.
• Integrated group sparsity to achieve neuron-level sparsity.
• Developed an efficient stochastic proximal gradient algorithm for optimization.

Main Results:

• The proposed method effectively promotes both connection-level and neuron-level sparsity.
• Redundant connections and unnecessary neurons were simultaneously removed.
• Experiments demonstrated the effectiveness of the approach on public datasets.

Conclusions:

• This work presents the first non-convex regularizer for simultaneous connection and neuron sparsity in DNNs.
• The method offers an efficient way to optimize DNNs for reduced computational and memory footprints.
• The proposed technique enhances DNN applicability in resource-limited settings.