Improving generalization of deep neural networks by leveraging margin distribution

Area of Science:

• Machine Learning
• Deep Learning Theory
• Computational Statistics

Background:

• Margin theory is increasingly used to understand deep neural network (DNN) generalization.
• Existing research primarily focuses on the spectrally-normalized minimum margin.
• This focus overlooks crucial information within the entire margin distribution.

Purpose of the Study:

• To develop a more comprehensive generalization bound for DNNs.
• To highlight the importance of margin distribution statistics beyond the minimum margin.
• To introduce and validate a novel method for controlling network complexity via margin ratio optimization.

Main Methods:

• Derived a generalization upper bound based on the statistics of the entire margin distribution.
• Introduced the margin ratio (margin standard deviation to expected margin) as a key complexity measure.
• Utilized a convex margin distribution loss function for DNNs to optimize the margin ratio.

Main Results:

• The proposed generalization bound is dominated by margin distribution statistics.
• Optimizing the margin ratio effectively controls DNN complexity and improves generalization.
• Experimental results and visualizations confirm the theoretical findings and the correlation between generalization gap and margin ratio.

Conclusions:

• The margin ratio is a critical factor for controlling DNN generalization performance.
• Focusing on the entire margin distribution offers a more nuanced understanding of generalization.
• The proposed method provides a practical approach to enhance DNN generalization by optimizing margin characteristics.