Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models.

Area of Science:

• Machine Learning
• Statistical Physics
• Deep Learning

Background:

• The bias-variance trade-off is a core concept in supervised learning, traditionally assuming optimal performance at intermediate model complexity.
• Deep Learning's success with over-parameterized models, which exceed training data capacity, contradicts classical understanding.
• Understanding bias and variance in these complex models is a critical research area.

Purpose of the Study:

• To derive analytic expressions for bias and variance in minimal over-parameterized models.
• To investigate the behavior of bias and variance beyond the classical trade-off.
• To provide a holistic understanding of generalization error in modern machine learning.

Main Methods:

• Utilizing statistical physics methods to analyze bias and variance.
• Developing minimal models for over-parameterization: linear regression and two-layer neural networks.
• Disentangling model architecture effects from random data sampling.

Main Results:

• Identified a phase transition in over-parameterized models where training error vanishes and test error initially diverges due to variance.
• Demonstrated that beyond this threshold, both bias and variance decrease in two-layer neural networks, improving test error.
• Showcased that over-parameterized models can overfit without noise and exhibit bias even with matching student-teacher models.

Conclusions:

• Classical bias-variance trade-offs do not fully apply to over-parameterized Deep Learning models.
• Over-parameterization can lead to improved generalization through simultaneous reduction of bias and variance.
• The study offers new insights into generalization error and the dynamics of bias and variance in complex machine learning models.