Large-Scale Heteroscedastic Regression via Gaussian Process.

Area of Science:

• Machine Learning
• Statistical Modeling
• Nonparametric Bayesian Methods

Background:

• Heteroscedastic regression models varying observation noise, crucial for machine learning and statistics.
• Standard heteroscedastic Gaussian process (HGP) regression offers strong performance but faces cubic time complexity, limiting its use with large datasets.
• Scalability is a key challenge for applying advanced HGP models to big data.

Purpose of the Study:

• To develop scalable inference algorithms for heteroscedastic Gaussian process (HGP) regression.
• To enhance the efficiency and capability of HGP models for large-scale machine learning applications.
• To overcome the computational limitations of traditional HGP methods.

Main Methods:

• Developed a variational sparse inference algorithm (VSHGP) for large-scale HGP regression.
• Introduced stochastic VSHGP (SVSHGP) using a factorized evidence lower bound for efficient stochastic variational inference.
• Proposed distributed VSHGP (DVSHGP) employing the Bayesian committee machine formalism and hybrid parameters for distributed computation and improved generalization.

Main Results:

• VSHGP effectively handles large datasets by reducing computational complexity.
• SVSHGP demonstrates enhanced efficiency through stochastic variational inference.
• DVSHGP shows superior scalability and capability by distributing computations and employing hybrid parameters, outperforming existing scalable HGP and homoscedastic GP methods on diverse datasets.

Conclusions:

• The developed VSHGP, SVSHGP, and DVSHGP algorithms significantly improve the scalability of heteroscedastic Gaussian process regression.
• These methods enable the application of advanced Bayesian nonparametric models to big data problems with varying noise levels.
• DVSHGP and SVSHGP offer a robust and efficient solution for scalable heteroscedastic regression, outperforming prior approaches.