Rethinking density ratio estimation based hyper-parameter optimization.

Area of Science:

• Machine Learning
• Computational Science

Background:

• Hyper-parameter optimization (HPO) is crucial for enhancing machine learning model performance.
• Current HPO methods often use binary classifiers for density-ratio estimation, which can overlook critical search space regions.

Purpose of the Study:

• To propose an improved HPO method utilizing multi-class classifiers.
• To address limitations of existing methods in capturing search space nuances and highlighting important regions.

Main Methods:

• Developed a novel HPO technique employing multi-class classifiers for ratio estimation.
• Partitioned hyper-parameter search spaces using classifier decision boundaries for finer analysis.
• Defined a new acquisition function weighting multi-class posterior probabilities based on sample distribution.

Main Results:

• The proposed method demonstrated superior performance compared to existing approaches.
• Achieved significant improvements in immediate regrets.
• Showcased enhanced convergence speed in experimental tasks.

Conclusions:

• Multi-class classifiers offer richer insights into hyper-parameter distributions than binary classifiers.
• The novel acquisition function effectively prioritizes informative regions in the search space.
• This method represents a significant advancement in efficient and effective hyper-parameter optimization.