Related Experiment Videos
Random Sparse Networks Training with Sharpness-Aware Regularization
IEEE Transactions on Pattern Analysis and Machine Intelligence
|August 10, 2026
Summary
The Dual Lottery Ticket Hypothesis (DLTH) suggests any random sparse subnetwork can be trained effectively. Our Sharpness-Aware Random Sparse Network Transformation (SA-RST) method validates this, improving sparse neural network training.
Area of Science:
- Artificial Intelligence
- Machine Learning
- Deep Learning
Background:
- Neural network optimization often relies on over-parameterization.
- Directly training sparse networks typically yields suboptimal performance.
- The Lottery Ticket Hypothesis (LTH) shows trainable sparse subnetworks exist within dense networks via pruning.
Purpose of the Study:
- To introduce the Dual Lottery Ticket Hypothesis (DLTH) for general sparse network training.
- To propose a method for enabling sparse subnetworks to achieve high trainability.
- To challenge the necessity of iterative pruning for discovering effective sparse networks.
Main Methods:
- Proposed the Dual Lottery Ticket Hypothesis (DLTH).
- Introduced Sharpness-Aware Random Sparse Network Transformation (SA-RST) for subnetwork transformation.
- Utilized a sharpness-aware regularization term to guide information extrusion from weights.
Main Results:
- Demonstrated the effectiveness of SA-RST in validating DLTH across CNN and Transformer architectures.
- Showcased improved trainability for randomly selected sparse subnetworks.
- Provided comprehensive comparisons with existing dynamic sparse training methods.
Conclusions:
- The Dual Lottery Ticket Hypothesis (DLTH) provides a new perspective on sparse network training.
- SA-RST effectively enables general training for sparse networks, challenging traditional LTH findings.
- This research offers significant insights into optimizing sparse deep learning models.
