Parameterized MDPs and Reinforcement Learning Problems-A Maximum Entropy Principle-Based Framework.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Sequential decision-making problems are prevalent in various domains.
• Traditional methods often struggle with noisy data and parameter uncertainty.
• Markov Decision Processes (MDPs) provide a mathematical framework for modeling these problems.

Purpose of the Study:

• To develop a robust framework for sequential decision-making.
• To enhance learning policies with robustness to noisy data.
• To determine unknown state and action parameters and perform sensitivity analysis.

Main Methods:

• The framework quantifies exploration using Shannon entropy of trajectories within MDPs.
• It determines stochastic policies maximizing entropy while ensuring low expected costs.
• The approach extends to parameterized MDPs and Reinforcement Learning (RL) problems.

Main Results:

• The proposed policy improves early exploration, leading to faster convergence and robust solutions.
• Demonstrated superior performance compared to Q-learning, Double Q-learning, and Soft Q-learning.
• Successfully applied to a 5G small cell network problem, optimizing routes and locations.

Conclusions:

• The framework offers a robust and efficient approach to sequential decision-making.
• It effectively handles noisy data and parameter dependencies.
• Provides valuable sensitivity measures for practical applications.