LOss-Based SensiTivity rEgulaRization: Towards deep sparse neural networks.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Science

Background:

• Neural network training often results in dense, computationally expensive models.
• Model compression techniques are crucial for efficient deployment of deep learning models.
• Existing methods may require extensive pre-training or complex rewinding procedures.

Purpose of the Study:

• To introduce a novel method for training sparse neural networks from scratch.
• To develop a regularization technique that promotes network sparsity.
• To evaluate the effectiveness of the proposed method in terms of compression and computational overhead.

Main Methods:

• Implemented LOBSTER (LOss-Based SensiTivity rEgulaRization) for neural network training.
• Defined parameter sensitivity as the impact of parameter variation on the loss function.
• Shrunk and pruned low-sensitivity parameters to achieve network sparsification.
• Trained networks from scratch without preliminary learning or rewinding.

Main Results:

• Achieved competitive compression ratios across various architectures and datasets.
• Demonstrated minimal computational overhead compared to existing methods.
• Successfully trained sparse neural networks with a desired topology.
• Validated the effectiveness of loss-based sensitivity for regularization.

Conclusions:

• LOBSTER offers an efficient approach to training sparse neural networks.
• The method enables significant model compression with minimal computational burden.
• Loss-based sensitivity regularization is a viable strategy for network sparsification.