Surrogate Modeling for Bayesian Optimization Beyond a Single Gaussian Process.

Area of Science:

• Machine Learning
• Optimization
• Artificial Intelligence

Background:

• Bayesian optimization (BO) is effective for expensive black-box functions but often relies on pre-selected Gaussian process (GP) kernels.
• Existing methods require domain knowledge for kernel selection, limiting adaptability.

Purpose of the Study:

• To develop a novel Bayesian optimization framework that adaptively selects surrogate models on-the-fly.
• To enhance the expressiveness and efficiency of BO for complex, expensive-to-evaluate functions.

Main Methods:

• Leveraging an ensemble of Gaussian processes (EGP) to create a GP mixture posterior.
• Employing Thompson sampling (TS) for adaptive acquisition function optimization.
• Utilizing random feature-based kernel approximation for scalability and parallel operation.

Main Results:

• The proposed EGP-TS method demonstrates enhanced expressiveness and adaptivity in surrogate modeling.
• Scalability is achieved through random feature approximation, enabling parallel computations.
• Theoretical analysis using Bayesian regret confirms convergence to the global optimum.

Conclusions:

• The EGP-TS framework offers a robust and efficient approach to Bayesian optimization, overcoming limitations of single-GP models.
• The method shows significant promise in diverse applications such as hyperparameter tuning, drug discovery, and robotics.