Sparse Variational Information Bottleneck Gaussian Processes for Uncertainty Estimation.

Area of Science:

• Machine Learning
• Statistical Modeling
• Information Theory

Background:

• Inducing-point sparse variational approximation (SVGP) scales Gaussian processes but struggles with accurate noise and variance estimation.
• Parametric predictive Gaussian process regressors (PPGPR) address point-wise uncertainty but have limitations with sparse methods.
• Existing methods utilize the information bottleneck (IB) principle sub-optimally, impacting uncertainty modeling.

Purpose of the Study:

• To re-examine uncertainty estimation in Gaussian process models through an information-theoretic lens.
• To identify and rectify sub-optimal applications of the information bottleneck principle in SVGP and PPGPR.
• To develop an improved Gaussian process method for enhanced uncertainty quantification and heteroskedastic noise modeling.

Main Methods:

• Analyzed SVGP and PPGPR through the information bottleneck (IB) principle, identifying limitations in utilizing latent function variance and noise estimation.
• Proposed a novel method, sparse variational information bottleneck Gaussian process (SVIBGP), by decomposing mutual information.
• Designed two coupled decoders within the SVIBGP framework to improve information processing and uncertainty estimation.

Main Results:

• Demonstrated that both SVGP and PPGPR employ the IB principle sub-optimally, leading to issues with uncertainty estimation.
• SVIBGP effectively utilizes input-dependent latent function variance for uncertainty modeling.
• Experiments show SVIBGP accurately accounts for heteroskedastic noise and provides superior uncertainty estimates compared to existing methods.

Conclusions:

• The proposed sparse variational information bottleneck Gaussian process (SVIBGP) method offers significant improvements in uncertainty estimation for large-scale Gaussian process modeling.
• SVIBGP successfully addresses the limitations of prior sparse variational methods by optimizing the information bottleneck principle.
• The method's ability to handle heteroskedastic noise makes it suitable for complex, real-world datasets.