Gradient-Based Framework for Bilevel Optimization of Black-Box Functions: Synergizing Model-Free Reinforcement...

Area of Science:

• Optimization
• Machine Learning
• Control Theory

Background:

• Bilevel optimization problems present significant challenges due to intricate variable interactions.
• Existing methods often oversimplify or lack scalability for high-dimensional, non-convex problems.
• Gradient-based approaches are hindered by implicit variable relationships and differentiability issues.

Purpose of the Study:

• To develop a gradient-based framework for bilevel optimization with black-box objectives.
• To enable scalable solutions for high-dimensional bilevel problems.
• To address challenges in differentiating implicit relationships in bilevel optimization.

Main Methods:

• Leveraging the implicit function theorem for gradient calculation.
• Employing model-free reinforcement learning (RL) for gradient-based updates.
• Utilizing policy gradient RL for scalable, high-dimensional updates.

Main Results:

• The framework successfully calculates upper-level objective gradients via implicit differentiation.
• Policy gradient RL facilitates scalable gradient-based updates for upper-level decisions.
• Demonstrated effectiveness in learning model predictive control policies for uncertain systems.

Conclusions:

• The proposed framework offers a scalable and efficient solution for bilevel problems with black-box objectives.
• Synergizes derivative-free optimization and implicit differentiation for enhanced performance.
• Opens new research avenues for complex optimization tasks.