Efficient Optimization for Sparse Gaussian Process Regression.

Area of Science:

• Machine Learning
• Statistical Modeling
• Computational Science

Background:

• Gaussian processes are powerful non-parametric models for regression and classification.
• Sparse Gaussian process regression aims to reduce computational complexity by selecting a subset of data points as inducing variables.
• Existing methods for selecting inducing sets often involve separate optimization objectives or are limited to continuous, differentiable domains.

Purpose of the Study:

• To propose an efficient, unified optimization algorithm for selecting inducing sets in sparse Gaussian process regression.
• To address limitations of previous methods, particularly their applicability to discrete domains and non-differentiable kernels.
• To enable simultaneous optimization of inducing sets and hyperparameters using a single objective function.

Main Methods:

• Developed a novel algorithm that estimates the inducing set and hyperparameters concurrently.
• The algorithm optimizes a single objective, which can be either the marginal likelihood or a variational free energy.
• Achieved linear space and time complexity with respect to the training set size.

Main Results:

• Demonstrated state-of-the-art performance for sparse Gaussian process regression in discrete input domains.
• Achieved competitive prediction accuracy in continuous input domains.
• Showcased a favorable trade-off between training and testing time, making it suitable for large-scale problems.

Conclusions:

• The proposed algorithm offers an efficient and versatile approach to sparse Gaussian process regression.
• It overcomes limitations of prior methods, extending applicability to discrete domains and non-differentiable kernels.
• The method provides a practical solution for large-scale regression problems with improved computational efficiency.