Online Calibrated and Conformal Prediction Improves Bayesian Optimization.

Area of Science:

• Machine Learning
• Sequential Decision-Making
• Optimization

Background:

• Accurate uncertainty quantification is vital for sequential model-based decision-making, particularly in Bayesian optimization.
• Model assumptions (e.g., Gaussianity) can be violated, leading to imperfect uncertainty estimates.
• Calibration, where predictive intervals accurately reflect outcome probabilities, is essential but challenging with non-stationary, action-dependent data.

Purpose of the Study:

• To investigate the role and necessity of specific uncertainties in model-based decision-making and Bayesian optimization.
• To introduce and validate a method for maintaining uncertainty calibration under non-i.i.d. data conditions.
• To demonstrate the practical benefits of calibrated Bayesian optimization in terms of convergence speed and final solution quality.

Main Methods:

• The study analyzes the requirements for uncertainty estimates in decision-making frameworks.
• It proposes novel algorithms based on online learning to ensure provable calibration maintenance.
• These algorithms are integrated into Bayesian optimization with minimal computational overhead.

Main Results:

• The proposed online learning algorithms provably maintain calibration on non-i.i.d. data.
• Calibrated Bayesian optimization demonstrates faster convergence to better optima compared to standard methods.
• Empirical validation shows improved performance on benchmark functions and hyperparameter optimization tasks.

Conclusions:

• Calibrated uncertainty estimates are essential for robust sequential decision-making and Bayesian optimization.
• Online learning provides an effective mechanism for maintaining calibration in dynamic environments.
• The proposed calibrated Bayesian optimization approach offers significant practical advantages for optimization problems.