Biased Dropout and Crossmap Dropout: Learning towards effective Dropout regularization in convolutional neural...

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks with many parameters are prone to overfitting.
• Dropout is a common regularization technique, but its impact on Convolutional Neural Networks (CNNs) requires further exploration.
• Standard Dropout can increase model training time.

Purpose of the Study:

• To introduce novel Dropout extensions, Biased Dropout and Crossmap Dropout, for CNNs.
• To address overfitting and improve generalization in deep learning models.
• To reduce the convergence time during the training phase of Dropout models.

Main Methods:

• Biased Dropout: Divides hidden units into two groups based on activation magnitude, applying different Dropout rates.
• Crossmap Dropout: Extends Dropout for convolution layers by applying a shared Dropout mask across feature maps to maintain correlations.
• Experiments were conducted on various benchmark datasets.

Main Results:

• Both Biased Dropout and Crossmap Dropout demonstrated superior generalization compared to standard Dropout.
• Biased Dropout achieved faster convergence during training.
• The proposed methods effectively regularize deep neural networks.

Conclusions:

• Biased Dropout and Crossmap Dropout are effective regularization techniques for CNNs.
• Appropriate noise assignment in hidden units can lead to effective regularization and faster training.
• These novel approaches offer improved performance and efficiency in deep learning models.