Automatic Sparse Connectivity Learning for Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Sparse neural networks offer potential for reduced computational load by eliminating zero weights.
• Existing pruning methods often require manual design choices and hyperparameters.
• Optimizing network connectivity is crucial for efficient deep learning.

Purpose of the Study:

• To introduce Sparse Connectivity Learning (SCL), a novel automatic pruning method for neural networks.
• To enable efficient reduction of computational resources and floating-point operations (FLOPs).
• To enhance network performance through optimized sparse connectivity.

Main Methods:

• Reparameterizing weights using a trainable variable and a binary mask.
• Utilizing a straight-through estimator (STE) with a focus on positive proxy gradients for mask optimization.
• Employing an identity STE for discrete mask relaxation and normalizing mask gradients for balanced training.
• Incorporating network connection count as a regularization term in the objective function.

Main Results:

• SCL automatically learns and selects critical network connections across various architectures.
• Experimental results show SCL outperforms existing methods in sparsity, accuracy, and FLOPs reduction.
• The method achieves optimized sparse connectivity without layer-specific pruning criteria or hyperparameters.

Conclusions:

• SCL provides an effective automatic pruning solution, overcoming limitations of prior approaches.
• The proposed method facilitates significant improvements in deep learning model efficiency and performance.
• SCL enables exploration of a larger hypothesis space for optimized network structures.