Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural...

Area of Science:

• Machine Learning
• Statistical Mechanics
• Deep Learning Theory

Background:

• Generalization error in overparameterized models like deep networks is not well understood.
• Kernel regression offers insights into infinitely overparameterized neural networks.

Purpose of the Study:

• To derive a theoretical understanding of generalization error in kernel regression.
• To provide an analytical expression for generalization error applicable to diverse kernels and data distributions.

Main Methods:

• Utilizing techniques from statistical mechanics.
• Deriving an analytical expression for generalization error.

Main Results:

• Developed a theory applicable to any kernel and data distribution.
• Applied the theory to real and synthetic datasets, including kernels from deep networks.
• Demonstrated that increased data can degrade generalization for noisy or incompatible data, causing non-monotonic learning curves.

Conclusions:

• The derived theory elucidates the inductive bias of kernel regression.
• Kernel compatibility with learning tasks can be characterized.
• Non-monotonic learning curves arise from noisy data or kernel incompatibility.