Regularizing Deep Neural Networks by Enhancing Diversity in Feature Extraction.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Oversized deep neural network models often generate redundant features, leading to inefficient processing.
• Feature redundancy can manifest as similar or shifted versions of existing features, causing unnecessary computations.

Purpose of the Study:

• To propose an efficient regularization technique for deep learning models.
• To address the issue of redundant feature extraction in neural networks.
• To improve model generalization and reduce overfitting through feature correlation analysis.

Main Methods:

• Developed a novel regularization technique based on feature correlations.
• Implemented an adaptive feature dropout mechanism to eliminate redundancy.
• Utilized relative cosine distances to differentiate and regularize correlated features (both positive and negative).

Main Results:

• Demonstrated avoidance of redundant feature extraction.
• Achieved networks that extract dissimilar features.
• Showcased reduced overfitting and improved generalization capabilities.
• Validated the technique across various deep learning architectures (MLP, CNN, SAE, GRU, LSTM) and datasets (MNIST, CIFAR-10, ImageNet, SNLI).

Conclusions:

• The proposed feature correlation-based regularization effectively reduces redundancy in deep neural networks.
• This approach leads to enhanced model performance, including better generalization and less overfitting.
• The technique is broadly applicable across diverse deep learning architectures and challenging datasets.