Reconciling modern machine-learning practice and the classical bias-variance trade-off

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Science

Background:

• The traditional bias-variance trade-off suggests models should balance underfitting and overfitting.
• Modern machine learning, particularly with complex models like neural networks, often interpolates data, seemingly defying this trade-off.
• This discrepancy raises questions about the foundational understanding and practical application of machine learning.

Purpose of the Study:

• To reconcile the classical bias-variance trade-off with the observed behavior of modern machine learning models.
• To introduce and validate a unified performance curve that explains the success of highly complex models.
• To provide a theoretical framework with practical implications for machine learning theory and practice.

Main Methods:

• Introduced a unified performance curve, termed the "double-descent" curve.
• Provided empirical evidence for the double-descent phenomenon across various models and datasets.
• Proposed a mechanism explaining the emergence of double descent.

Main Results:

• Demonstrated that increasing model capacity beyond data interpolation can lead to improved predictive accuracy.
• Showcased the ubiquity of the double-descent curve across diverse machine learning scenarios.
• Established a connection between model performance and its structural properties.

Conclusions:

• The classical bias-variance trade-off is a limited view; the double-descent curve offers a more comprehensive understanding.
• Highly complex, interpolating models can achieve superior performance, contrary to classical expectations.
• This work bridges theoretical gaps and offers practical insights for developing and understanding advanced machine learning systems.