LocalDrop: A Hybrid Regularization for Deep Neural Networks

Area of Science:

• Machine Learning
• Deep Learning
• Neural Network Regularization

Background:

• Overfitting is a significant challenge in neural network training.
• Existing regularization techniques aim to mitigate this issue.
• Local Rademacher complexity offers a theoretical framework for analyzing generalization error.

Purpose of the Study:

• To introduce LocalDrop, a new regularization algorithm for neural networks.
• To develop a novel regularization function based on local Rademacher complexity.
• To demonstrate the effectiveness of LocalDrop in improving model generalization.

Main Methods:

• Derived an upper bound for local Rademacher complexity through mathematical deduction.
• Developed a new regularization function applicable to fully-connected networks (FCNs) and convolutional neural networks (CNNs).
• Proposed a two-stage procedure for optimizing keep rate and weight matrices.

Main Results:

• The proposed regularization function effectively reduces overfitting in FCNs and CNNs.
• LocalDrop demonstrated superior performance compared to existing regularization algorithms in extensive experiments.
• Analysis of dropout and DropBlock in FCNs and CNNs, respectively, was conducted.

Conclusions:

• LocalDrop provides an effective strategy for regularizing neural networks.
• The method shows promise for enhancing the generalization capabilities of various neural network architectures.
• Further research can explore hyperparameter tuning for optimal performance.