Randomness Regularization With Simple Consistency Training for Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Randomness, such as dropout, is commonly used in neural network training to aid optimization and prevent overfitting.
• However, dropout introduces inconsistencies between training and inference phases, potentially impacting model reliability.

Purpose of the Study:

• To introduce R-Drop, a simple consistency training strategy designed to regularize randomness in neural networks.
• To address the training-inference inconsistency caused by dropout-based regularization methods.

Main Methods:

• R-Drop enforces consistency between two output distributions sampled via dropout-based randomness for each training sample.
• It minimizes the bidirectional KL-divergence between these output distributions.
• Theoretical analysis supports R-Drop's ability to reduce sub-structure inconsistencies and bridge the gap between full and sub-model losses.

Main Results:

• Experiments across 7 deep learning tasks and 23 datasets show R-Drop's universal effectiveness.
• The method benefits feed-forward, recurrent, and graph neural networks under supervised, parameter-efficient, and semi-supervised learning.
• R-Drop achieved state-of-the-art results on WMT14 English → German and English → French translation tasks using a vanilla Transformer model.

Conclusions:

• R-Drop offers a simple yet effective approach to regularize randomness in deep learning.
• It successfully mitigates training-inference inconsistencies, leading to improved performance and state-of-the-art results on challenging translation tasks.