Hyperparameter Optimization of Non-linear Machine Learning Models Using Bi-level Data-Driven Optimization.

Area of Science:

• Machine Learning
• Optimization
• Computational Science

Background:

• Hyperparameter tuning is crucial for machine learning (ML) model performance but often relies on inefficient trial-and-error methods.
• Conventional techniques like grid search, random search, and Bayesian optimization can be exhaustive and computationally expensive.
• Existing methods struggle to capture the complex interdependence between hyperparameter selection and model evaluation.

Purpose of the Study:

• To present a systematic and data-driven decision-making approach for tuning ML model hyperparameters using cross-validation.
• To offer a more efficient and precise alternative to conventional hyperparameter optimization methods.
• To demonstrate the enhancement of bi-level data-driven optimization, particularly with non-linear loss functions.

Main Methods:

• Formulating hyperparameter tuning as a bi-level optimization problem.
• Utilizing the Data-driven Optimization of bi-level Mixed-Integer non-linear problems (DOMINO) framework to approximate the bi-level problem as a single-level problem.
• Evaluating 17 diverse data-driven optimization algorithms (heuristic, deterministic, local, global, sample-based, model-based) on regression and classification tasks.

Main Results:

• The data-driven bi-level approach significantly outperforms conventional tuning algorithms in predictive accuracy on unseen data.
• ML models tuned with this method exhibit superior generalization capabilities across various tasks.
• Local optimization algorithms within DOMINO are efficient for single hyperparameters, while global algorithms excel with multiple or non-linear hyperparameters.

Conclusions:

• Bi-level data-driven optimization, implemented via the DOMINO framework, provides a powerful and efficient strategy for hyperparameter tuning.
• This approach enhances the accuracy and generalization of machine learning models, especially in complex scenarios.
• The choice between local and global optimization algorithms within the framework depends on the model's hyperparameter complexity and characteristics.