Bias-variance decomposition of overparameterized regression with random linear features.

Area of Science:

• Statistical learning theory
• Machine learning
• Computational physics

Background:

• The classical bias-variance trade-off dictates optimal model complexity for prediction.
• Overparameterized models challenge this trade-off, achieving high performance despite numerous parameters.
• Understanding the behavior of overparameterized models is crucial for advancing predictive modeling.

Purpose of the Study:

• To analyze the bias-variance trade-off in a simple overparameterized model: regression with random linear features.
• To derive analytic expressions for key performance metrics including training error, test error, bias, and variance.
• To investigate the phase transitions and their underlying mechanisms in this model.

Main Methods:

• Zero-temperature cavity method for deriving analytic expressions.
• Random matrix theory to analyze eigenvalue properties of the Hessian matrix.
• Comparison with random nonlinear features and ordinary regression models.

Main Results:

• Identified three phase transitions in the linear random features model.
• Observed two transitions into an interpolation regime (zero training error).
• Found an additional transition between high-bias and low-bias regimes.
• Linked phase transitions to small non-zero eigenvalues in the Hessian matrix.

Conclusions:

• The linear random features model exhibits complex phase transitions not captured by classical bias-variance trade-off.
• Linear basis functions introduce unique phase transition behaviors compared to nonlinear models.
• This study provides fundamental insights into the statistical mechanics of overparameterized models.