To understand double descent, we need to understand VC theory

Area of Science:

• Machine Learning
• Theoretical Computer Science
• Statistical Learning Theory

Background:

• Deep learning models exhibit 'double descent,' fitting training data perfectly while generalizing well.
• Current understanding suggests VC-theory cannot explain this phenomenon.
• Over-parameterized models challenge traditional generalization bounds.

Purpose of the Study:

• To reconcile the 'double descent' phenomenon with VC-theoretical concepts.
• To demonstrate that VC-dimension and Structural Risk Minimization can explain deep learning generalization.
• To provide a theoretical framework for understanding generalization curves in various data scenarios.

Main Methods:

• Analysis of generalization performance within the VC-theoretical framework.
• Application of VC-dimension and Structural Risk Minimization principles.
• Empirical validation using classical VC-generalization bounds.
• Investigation of transfer learning generalization with pre-trained networks.

Main Results:

• Double descent is explainable by VC-theoretical concepts.
• VC-generalization bounds accurately model double descent generalization curves.
• The analysis provides insights into generalization across diverse datasets (e.g., high-dimensional, noisy).

Conclusions:

• VC-theory offers a robust framework for understanding deep learning generalization.
• The proposed analysis enhances comprehension of generalization curves for various data types.
• This work bridges the gap between theoretical guarantees and practical deep learning performance.