SURPRISES IN HIGH-DIMENSIONAL RIDGELESS LEAST SQUARES INTERPOLATION

Area of Science:

• Machine Learning
• Statistical Learning Theory

Background:

• Interpolators, which achieve zero training error, are increasingly important in machine learning.
• State-of-the-art neural networks often function as interpolators.
• Understanding interpolator behavior is key to advancing machine learning.

Purpose of the Study:

• To investigate minimum L2 norm (ridgeless) interpolation least squares regression.
• To analyze interpolator behavior in the high-dimensional regime (p ~ n).
• To quantitatively explain phenomena observed in large-scale neural networks and kernel machines.

Main Methods:

• Studied minimum L2 norm interpolation regression.
• Analyzed two feature distribution models: linear and nonlinear (one-layer neural network).
• Focused on the high-dimensional setting where parameters (p) are similar to samples (n).

Main Results:

• Quantitatively recovered phenomena seen in large neural networks and kernel methods.
• Observed the 'double descent' behavior in prediction risk.
• Demonstrated the benefits of overparametrization in interpolating models.

Conclusions:

• Minimum L2 norm interpolation regression exhibits complex behaviors in high dimensions.
• The study provides theoretical insights into why overparameterized models like neural networks perform well.
• Findings help explain the success of modern machine learning architectures.