An Optimal Transport Analysis on Generalization in Deep Learning.

Area of Science:

• Machine Learning
• Deep Learning Theory
• Optimal Transport

Background:

• Deep neural networks (DNNs) excel in various tasks but their generalization ability, despite overparameterization, remains theoretically unexplained.
• Traditional generalization error analysis fails for DNNs due to loose worst-case bounds on large parameter spaces.

Purpose of the Study:

• To propose a novel optimal transport-based framework for analyzing DNN generalization.
• To explain why deeper neural networks generalize better than shallower ones.

Main Methods:

• Developed an average-case analysis of generalization error, dependent on learning algorithms and data distributions.
• Derived upper bounds on generalization error using algorithmic transport cost (Wasserstein distance).
• Investigated generalization error bounds using total variation distance, relative entropy, and Vapnik-Chervonenkis (VC) dimension.

Main Results:

• Generalization error is bounded by the algorithmic transport cost.
• Established conditions for loss functions to bound generalization error using probability metrics.
• Demonstrated that generalization error in DNNs decreases exponentially with increasing network depth.

Conclusions:

• Optimal transport provides a practical and accurate framework for understanding DNN generalization.
• The proposed theory explains the superior generalization of deeper networks.
• This work offers new theoretical insights into deep learning generalization.