Shakeout: A New Approach to Regularized Deep Neural Network Training.

Area of Science:

• Machine Learning
• Deep Learning
• Artificial Intelligence

Background:

• Deep neural networks (DNNs) have achieved significant success in various applications.
• Regularization techniques like Dropout are crucial for training effective DNNs by preventing overfitting.
• Dropout randomly discards units during training, impacting model generalization.

Purpose of the Study:

• Introduce Shakeout, a novel regularization approach for DNNs.
• Compare Shakeout's performance against Dropout in image classification.
• Investigate Shakeout's impact on weight sparsity, feature grouping, and training stability.

Main Methods:

• Developed Shakeout, a method that randomly enhances or reverses unit contributions instead of discarding them.
• Conducted classification experiments using deep architectures on MNIST, CIFAR-10, and ImageNet datasets.
• Analyzed weight sparsity, grouping effects, and training stability under supervised and unsupervised settings.

Main Results:

• Shakeout effectively combats overfitting and surpasses Dropout's performance in image classification.
• Empirical evidence shows Shakeout promotes sparser weights and a grouping effect among input units.
• Shakeout demonstrates superiority over Dropout for deep model compression and reduces training instability.

Conclusions:

• Shakeout offers an effective alternative to Dropout for regularizing deep neural networks.
• The method's ability to induce sparsity and grouping makes it valuable for model compression.
• Shakeout contributes to more stable and robust deep learning model training.