Generalization and Expressivity for Deep Nets.

Area of Science:

• Machine Learning
• Theoretical Computer Science
• Artificial Intelligence

Background:

• Deep learning's practical success necessitates theoretical understanding.
• Generalization and expressivity are key metrics for deep neural network (DNN) behavior.
• Existing research often focuses on either expressivity (large capacity) or generalization (small capacity).

Purpose of the Study:

• To investigate the theoretical advantages of deep nets by considering both expressivity and generalization.
• To explore if deep architectures can achieve high expressivity without a significant increase in capacity compared to shallow networks.
• To derive theoretical learning rates for deep nets.

Main Methods:

• Constructed a two-hidden-layer deep neural network.
• Evaluated expressivity using localized and sparse approximation.
• Employed the covering number to measure network capacity.
• Derived learning rates for empirical risk minimization.

Main Results:

• Demonstrated that deep nets possess excellent expressive power (localized and sparse approximation) without substantially increasing capacity compared to shallow nets.
• Showcased that deep architectures can achieve high expressivity and generalization simultaneously.
• Established near-optimal learning rates for deep nets, theoretically validating their advantages.

Conclusions:

• Deep neural networks offer theoretical advantages in both expressivity and generalization.
• The findings provide a learning theory perspective on why deep learning models are effective.
• This work bridges the gap between expressivity and generalization in deep learning theory.