SSGD: SPARSITY-PROMOTING STOCHASTIC GRADIENT DESCENT ALGORITHM FOR UNBIASED DNN PRUNING.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Deep neural networks (DNNs) achieve high performance but are often over-parameterized, leading to computational inefficiency.
• Parameter redundancy in DNNs necessitates methods for creating more compact and efficient models.
• Sparse signal recovery (SSR) techniques offer solutions for finding compact representations in overcomplete linear problems.

Purpose of the Study:

• To explore the application of iterative reweighting methods from SSR to learn efficient DNNs.
• To develop a method for creating sparse networks with reduced computation and storage requirements.
• To propose a novel framework for learning sparse connections within DNN architectures without biasing the optimization.

Main Methods:

• Application of iterative reweighting methods popular in SSR to DNNs.
• Development of a reweighting framework utilizing affine scaling transformation.
• Introduction of Sparsity-promoting Stochastic Gradient Descent (SSGD) with simple gradient-based updates.

Main Results:

• SSGD effectively promotes sparsity in deep neural networks.
• The proposed method achieves efficient network sparsification.
• SSGD outperforms existing methods on image classification tasks on MNIST and CIFAR-10 datasets.

Conclusions:

• Iterative reweighting methods from SSR can be successfully applied to learn efficient DNNs.
• SSGD provides a simple and implementable approach to network sparsification.
• The developed method offers a promising direction for creating computationally efficient deep learning models.