Conditional Deep Gaussian Processes: Empirical Bayes Hyperdata Learning.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Statistics

Background:

• Deep learning excels at feature extraction, while Gaussian Processes (GP) provide probabilistic modeling.
• Combining these offers expressive Bayesian learning, but deterministic feature extractors can overfit.
• Existing deep kernel learning methods use GPs with deep networks for feature extraction.

Purpose of the Study:

• To propose a conditional deep Gaussian process (DGP) model for enhanced Bayesian learning.
• To integrate Bayesian networks within a hierarchical GP structure for improved robustness.
• To leverage hyperdata as function supports within the DGP framework.

Main Methods:

• Developed a conditional DGP where intermediate GPs are supported by hyperdata.
• Utilized a moment matching approach to approximate the marginal prior for conditional DGP.
• Learned hyperdata by optimizing an approximate marginal likelihood, linking to empirical Bayes.
• Demonstrated equivalence to deep kernel learning with dense hyperdata.

Main Results:

• Conditional DGP offers more Bayesian inference compared to deep kernel learning.
• Preliminary results show enhanced expressive power through hierarchical depth and hyperdata learning.
• The model outperforms GP kernel composition, DGP variational inference, and deep kernel learning in extrapolation tasks.

Conclusions:

• Conditional DGP provides a more robust and Bayesian approach to combining deep learning and Gaussian processes.
• Hyperdata learning and hierarchical structure contribute to the model's expressive power.
• The proposed method offers advantages over existing techniques like deep kernel learning and variational inference.