Regularization of deep neural networks with spectral dropout

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Overfitting is a significant challenge in deep neural networks, hindering generalization.
• The 'Dropout' technique was a key innovation for addressing overfitting in Convolutional Neural Networks (CNNs).

Purpose of the Study:

• To introduce 'Spectral Dropout', a novel regularization technique for enhancing deep neural network generalization.
• To improve upon existing methods like Dropout and Drop-Connect in terms of efficiency and performance.

Main Methods:

• Spectral Dropout is implemented within CNN weight layers using a decorrelation transform with fixed basis functions.
• The method eliminates weak and noisy Fourier domain coefficients from neural network activations.

Main Results:

• Spectral Dropout significantly improves generalization ability, outperforming current regularization techniques.
• The method accelerates network convergence by approximately twofold compared to Dropout and Drop-Connect.
• It enables considerably higher neuron pruning rates, with an increase of around 30%.

Conclusions:

• Spectral Dropout offers an efficient and effective approach to combat overfitting in deep neural networks.
• The technique demonstrates superior performance and faster training convergence.
• Spectral Dropout can be combined with other regularization methods for further performance enhancements.