Lottery Jackpots Exist in Pre-Trained Models.

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Network pruning reduces model complexity but often requires extensive training or wide networks.
• Existing methods are computationally expensive and limit practical applications of network pruning.

Purpose of the Study:

• To identify high-performing, sparse sub-networks (lottery jackpots) within pre-trained models without weight retraining.
• To improve the efficiency of searching for these lottery jackpots.
• To analyze and optimize the lottery jackpot searching process.

Main Methods:

• Identifying sparse sub-networks (lottery jackpots) in pre-trained models without modifying weights.
• Leveraging magnitude-based pruning to initialize sparse masks, reducing search costs.
• Proposing a novel short restriction method to stabilize mask searching and improve convergence.

Main Results:

• A lottery jackpot sub-network with 10% of VGGNet-19 parameters achieved original performance on CIFAR-10 without weight retraining.
• Initializing with magnitude-based pruning reduced lottery jackpot search cost by at least 3x.
• A ResNet-50 lottery jackpot achieved >70% top-1 accuracy on ImageNet with 90% weight removal in only 5 search epochs.

Conclusions:

• High-performing sparse sub-networks (lottery jackpots) exist in pre-trained models, offering efficient network compression.
• Optimized search strategies, including magnitude-based initialization and short restriction, significantly enhance the efficiency of finding lottery jackpots.
• This approach enables substantial model compression with minimal performance loss, making network pruning more accessible.