A Model-based Framework for Black-box Problem Comparison Using Gaussian Processes

An important challenge in black-box optimization is to be able to understand the relative performance of different algorithms on problem instances. This challenge has motivated research in exploratory landscape analysis and algorithm selection, leading to a number of frameworks for analysis. However, these procedures often involve significant assumptions, or rely on information not typically available. In this paper we propose a new, model-based framework for the characterization of black-box optimization problems using Gaussian Process regression. The framework allows problem instances to be compared to each other in a relatively simple way. The model-based approach also allows us to assess the goodness of fit and Gaussian Processes lead to an efficient means of model comparison. The implementation of the framework is described and validated on several test sets as one benchmark problem is slowly transformed into another.