Less Is More: Adaptive Trainable Gradient Dropout for Deep Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Deep neural networks (DNNs) offer significant computational power but require vast datasets.
• Large datasets can hinder DNN generalization, necessitating regularization techniques like dropout.
• Existing methods like random weight dropout may not optimally enhance network sparsity.

Purpose of the Study:

• To propose a novel selective gradient dropout method for DNNs.
• To enhance network sparsity adaptively by freezing salient connections.
• To improve DNN generalization and reduce training time.

Main Methods:

• Implemented a selective gradient dropout technique that learns to freeze specific network connections.
• Focused on adaptive sparsity by prioritizing salient weights over random weight dropping.
• Evaluated the method on various image classification datasets.

Main Results:

• The proposed sparse network achieved superior performance compared to the baseline.
• The selective gradient dropout method significantly reduced the number of training epochs required.
• The technique adaptively increased network sparsity by utilizing more salient weights.

Conclusions:

• Selective gradient dropout is an effective regularization technique for DNNs.
• The method enhances both performance and training efficiency in image classification tasks.
• This approach offers a more targeted way to achieve sparsity and improve generalization.