Playing the Lottery With Concave Regularizers for Sparse Trainable Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Sparse neural networks offer reduced computational and storage needs during inference.
• The lottery ticket hypothesis (LTH) suggests effective sparse subnetworks can be trained in isolation.
• Finding these 'winning tickets' efficiently remains an open research challenge.

Purpose of the Study:

• To propose a novel class of methods for identifying sparse subnetworks ('winning tickets').
• To leverage concave regularization for promoting network topology sparsity.

Main Methods:

• Utilizing concave regularization to encourage sparsity in a relaxed binary mask representing network topology.
• Theoretical analysis within a convex framework to validate the method's effectiveness.
• Extensive numerical testing across diverse datasets and neural network architectures.

Main Results:

• The proposed method demonstrates improved performance compared to state-of-the-art algorithms.
• Concave regularization effectively promotes sparsity in network topology masks.
• Winning tickets identified by the method show strong isolated training capabilities.

Conclusions:

• The novel concave regularization approach provides an effective strategy for finding winning tickets in sparse neural networks.
• This method enhances the efficiency of both training and inference phases.
• The findings contribute to advancing the field of sparse deep learning.