Hyperparameter Optimization of Non-linear Machine Learning Models Using Bi-level Data-Driven Optimization

This study presents a systematic decision-making approach for tuning hyperparameters of machine learning (ML) models that employ the cross-validation technique in their learning process. It provides a more efficient and precise alternative to conventional hyperparameter optimization methods (i.e., grid search, random search, and Bayesian optimization) and demonstrates that bi-level data-driven optimization enhances this task, especially when non-linear loss functions are used in the training process. Hyperparameters are external adjustable parameters that configure and control the learning process of ML models and cannot be estimated directly from the training data. Their tuning plays a crucial role in constructing accurate and generalized ML models. However, this process is often treated as a trial-and-error search, where numerous hyperparameter combinations are evaluated through training and validation, and the best-performing one is selected after an exhaustive search. An alternative is to pose hyperparameter tuning as a bi-level optimization problem, which explicitly captures the interdependence between hyperparameter selection and model evaluation. This formulation enables a more structured optimization strategy but introduces significant algorithmic and computational challenges. To address this, we use the Data-driven Optimization of bi-level Mixed-Integer non-linear problems (DOMINO) framework to approximate the bi-level formulation of the cross-validated hyperparameter optimization problem as a single-level problem. This transformation enables the use of data-driven methodologies to solve the otherwise intractable bi-level problem more efficiently, while still capturing the unknown interactions between hyperparameter choices and model performance. We evaluate 17 different data-driven optimization algorithms, including heuristic vs. deterministic methods, local vs. global approaches, and sample-based vs. model-based algorithms, on six hyperparameter tuning problems for regression and classification tasks. Our results show that the data-driven bi-level approach outperforms conventional tuning algorithms in predictive accuracy on blind test datasets and yields ML models that exhibit excellent generalization across all case studies. We further observe that local optimization algorithms integrated into DOMINO are more computationally efficient when tuning models with a single hyperparameter, whereas global algorithms are more effective for models involving multiple hyperparameters or non-linear characteristics.