Maximum Relevance Minimum Redundancy Dropout with Informative Kernel Determinantal Point Process.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Deep neural networks excel in computer vision but are prone to overfitting.
• Standard dropout mitigates overfitting but can be suboptimal.
• Existing dropout improvements neglect task-relevant information and kernel diversity.

Purpose of the Study:

• To develop an efficient end-to-end dropout algorithm for selecting informative neurons.
• To address the challenge of dropping less informative neurons in deep learning.
• To improve neural network generalization and classification performance.

Main Methods:

• Proposed an efficient end-to-end dropout algorithm.
• Utilized determinantal point process (DPP) sampling for neuron diversity.
• Developed a mutual information (MI)-based merit function for task specificity.
• Introduced DPPMI dropout combining MI and DPP sampling.

Main Results:

• DPPMI dropout adaptively adjusts neuron retention rates based on task contribution.
• The method demonstrated superior performance over state-of-the-art dropout algorithms.
• Empirical studies on MNIST, SVHN, CIFAR10, and CIFAR100 confirmed effectiveness.

Conclusions:

• DPPMI dropout effectively identifies and retains informative neurons.
• The proposed method enhances neural network generalization and classification accuracy.
• This approach offers a significant advancement in deep learning regularization techniques.