A Fully Natural Gradient Scheme for Improving Inference of the Heterogeneous Multioutput Gaussian Process Model

Area of Science:

• Machine Learning
• Statistical Modeling
• Computational Statistics

Background:

• Multioutput Gaussian processes (GPs) are extended to model heterogeneous outputs using individual likelihood functions.
• A vector-valued GP prior with a linear model of coregionalization (LMC) covariance jointly models likelihood parameters as latent functions.
• Stochastic variational inference (SVI) is enabled via an inducing points' framework, yielding tractable variational bounds.

Purpose of the Study:

• To address optimization challenges in heterogeneous multioutput GP models, specifically the conditioning issues with adaptive gradient methods.
• To introduce a novel natural gradient (NG) optimization scheme using an exploratory distribution over hyperparameters.
• To extend the heterogeneous multioutput model with latent functions from convolution processes and optimize it using NG.

Main Methods:

• Developed a natural gradient (NG) optimization scheme by introducing an exploratory distribution over hyperparameters for joint inference.
• Extended the heterogeneous multioutput GP model to incorporate convolution processes for latent functions.
• Applied SVI to the convolutional model for scalability and optimized it using the proposed NG scheme.

Main Results:

• The NG optimization scheme achieved superior local optima and higher test performance rates compared to adaptive gradient methods for both LMC and convolution process models.
• Demonstrated the scalability of the convolutional GP model through SVI.
• Showcased the effectiveness of the NG optimization for the scalable convolutional model.

Conclusions:

• The proposed natural gradient optimization scheme significantly improves performance and stability for heterogeneous multioutput Gaussian processes.
• Convolution processes offer a flexible alternative for latent function modeling in heterogeneous multioutput GPs.
• The developed methods provide efficient and scalable solutions for complex multioutput GP modeling and inference.