High-Performance Hybrid-Global-Deflated-Local Optimization with Applications to Active Learning

Area of Science:

• * Computational Science and Engineering
• * Machine Learning and Artificial Intelligence

Background:

• * Mathematical optimization is crucial for science and industry, but faces a trade-off between evaluation count and optimum quality.
• * Machine learning and active learning require high-quality optima for accurate surrogate models, often complicated by missing offline data.
• * Current optimization methods can stall active learning due to sequential data collection and training.

Purpose of the Study:

• * To present a high-performance hybrid optimization algorithm (HGDL) for scientific and industrial applications.
• * To address the challenge of finding global or high-quality local optima in machine learning and active learning.
• * To improve the efficiency of autonomous experimentation by optimizing surrogate model training.

Main Methods:

• * Developed a hybrid global and local optimization algorithm (HGDL) combining derivative-free and derivative-based strategies.
• * Implemented redundancy avoidance by deflating the objective function around found optima.
• * Designed HGDL for asynchronous parallelism, running computationally intensive local optimizations concurrently on separate nodes.

Main Results:

• * HGDL yields an ordered list of unique local optima, mitigating redundancy.
• * The algorithm effectively utilizes parallelism for faster optimization.
• * Asynchronous operation allows immediate use of found solutions while continuing the search.

Conclusions:

• * HGDL offers a robust solution to the optimization challenges in machine learning and active learning.
• * The proposed strategy enhances autonomous experimentation through efficient surrogate model approximation.
• * This hybrid approach improves the balance between optimization speed and solution quality.