Multi-Objective Model Selection via Racing.

Area of Science:

• Machine Learning
• Computational Intelligence
• Optimization

Background:

• Model selection is crucial in machine learning, often involving multiple, competing objectives.
• Hyper-parameter tuning in multi-task learning exemplifies a multi-objective challenge requiring simultaneous optimization.

Purpose of the Study:

• To introduce S-Race, a novel multi-objective racing algorithm (RA) for efficient model selection.
• To reliably identify Pareto optimal models while minimizing computational cost.
• To address multi-objective model selection (MOMS) problems effectively.

Main Methods:

• Utilizes a racing algorithm (RA) approach to eliminate non-promising models early.
• Employs the nonparametric sign test for pair-wise dominance identification.
• Adopts a discrete Holm's step-down procedure to control family-wise error rate, with adaptive significance level adjustment.

Main Results:

• S-Race efficiently identifies Pareto optimal models in multi-objective scenarios.
• Demonstrates effectiveness and efficiency compared to brute-force approaches.
• Successfully applied to diverse MOMS problems including SVM selection, ABC algorithm tuning, and hybrid recommendation systems.

Conclusions:

• S-Race is an efficient and effective algorithm for multi-objective model selection.
• The nonparametric sign test and Holm's procedure provide robust dominance identification and error control.
• The algorithm shows versatility across various machine learning and optimization tasks.