Contrasting random and learned features in deep Bayesian linear regression.

Area of Science:

• Deep Learning Theory
• Machine Learning
• Artificial Intelligence

Background:

• Understanding feature learning's role in generalization is crucial for modern deep learning.
• Deep Bayesian linear neural networks offer a tractable model class for theoretical analysis.
• Generalization performance is influenced by model architecture, data characteristics, and training procedures.

Purpose of the Study:

• To investigate how feature learning affects generalization in deep Bayesian linear neural networks.
• To characterize the interplay between model architecture (width, depth), data density, and prior mismatch.
• To compare generalization behaviors of deep random feature models and fully trained deep networks.

Main Methods:

• Comparative analysis of deep random feature models and deep neural networks.
• Training on unstructured Gaussian data with varying label noise.
• Examination of samplewise and modelwise double-descent phenomena.
• Analysis of generalization curves and kernel-limit behavior.

Main Results:

• Both model types exhibit samplewise double descent with label noise.
• Random feature models can show modelwise double descent (with bottlenecks); deep networks do not.
• Optimal width strategies differ: specific widths for random features, extreme widths for deep networks.
• Kernel-limit learning curves cannot distinguish between the two model types.

Conclusions:

• Architectural details significantly impact generalization in deep regression models.
• Feature learning dynamics differ between random feature models and fully trained deep networks.
• Findings provide insights into the theoretical underpinnings of deep learning generalization.