Neural Network Gaussian Processes by Increasing Depth

Area of Science:

• Machine Learning
• Deep Learning Theory
• Gaussian Processes

Background:

• Growing interest in the connection between infinitely wide neural networks and Gaussian processes.
• Existing theories primarily focus on width-induced neural network Gaussian processes (NNGPs).
• Limited understanding of how neural network depth influences network behavior.

Purpose of the Study:

• To investigate if increasing neural network depth can also induce Gaussian process behavior.
• To theoretically characterize the properties of depth-induced Gaussian processes.
• To explore the practical implications of depth-induced Gaussian processes in deep learning.

Main Methods:

• Utilizing a shortcut network architecture inspired by width-depth symmetry.
• Theoretically analyzing the uniform tightness property of the depth-induced Gaussian process.
• Investigating the smallest eigenvalue of the Gaussian process kernel.
• Conducting regression experiments on benchmark datasets.

Main Results:

• Demonstrated that increasing neural network depth can lead to a Gaussian process.
• Provided theoretical characterizations of the depth-induced Gaussian process properties.
• Empirically validated the performance of the proposed Gaussian process through regression tasks.

Conclusions:

• The study extends the theory of neural network Gaussian processes by introducing depth as an inducing factor.
• Theoretical characterizations enhance the understanding of depth-induced Gaussian processes.
• Findings contribute to a more comprehensive view of deep learning behaviors and offer potential for future applications.