Cross-entropy optimization of control policies with adaptive basis functions

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Markov decision processes (MDPs) are fundamental to sequential decision-making under uncertainty.
• Finding optimal control policies in continuous-state, discrete-action MDPs remains a significant challenge.
• Existing methods often struggle with scalability and policy representation complexity.

Purpose of the Study:

• To introduce a novel algorithm for direct search of control policies in continuous-state discrete-action MDPs.
• To develop a flexible policy representation using adaptive basis functions (BFs).
• To optimize policy representation by tuning BFs and action assignments.

Main Methods:

• The proposed algorithm employs direct search for closed-loop policies.
• It utilizes a specified number of basis functions (BFs), with their type, number, locations, and shapes optimized.
• Optimization is performed using the cross-entropy method, evaluating policies via Monte Carlo simulations from representative initial states.

Main Results:

• The algorithm reliably obtains effective policies using a small number of BFs across problems with two to six state variables.
• Cross-entropy policy search with adaptive BFs demonstrates superior performance compared to value-function techniques.
• It significantly outperforms policy search using the DIRECT optimization algorithm.

Conclusions:

• The developed cross-entropy policy search with adaptive BFs offers an efficient and flexible approach for solving complex MDPs.
• This method requires substantially fewer basis functions than traditional value-function approaches.
• The algorithm shows strong potential for applications in various domains requiring optimal control policy discovery.