Guided Policy Exploration for Markov Decision Processes Using an Uncertainty-Based Value-of-Information Criterion

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Reinforcement learning (RL) faces challenges in large action-state spaces due to extensive exploration requirements.
• Conventional RL exploration strategies often employ stochastic methods, potentially leaving significant portions of the policy space unvisited.
• Inefficient policy space coverage can hinder effective learning and policy optimization.

Purpose of the Study:

• To develop an uncertainty-based, information-theoretic approach for guided stochastic searches in RL.
• To enhance policy space coverage and improve learning efficiency in complex environments.
• To introduce a method that optimally balances exploration costs and search granularity.

Main Methods:

• Proposed an uncertainty-based, information-theoretic approach leveraging the value of information (VoI).
• Incorporated a state-transition uncertainty factor to guide exploration into novel regions.
• Utilized policy cross-entropy for hyperparameter optimization to further enhance training rates.

Main Results:

• The uncertainty-based VoI policies demonstrated superior performance compared to traditional stochastic exploration strategies.
• The proposed method achieved better policies in fewer training episodes across evaluated games.
• Policy cross-entropy effectively guided hyperparameter selection, improving the training rate.

Conclusions:

• The uncertainty-based VoI approach offers a more effective method for exploring complex policy spaces in reinforcement learning.
• This strategy leads to improved policy performance and accelerated training.
• The integration of state-transition uncertainty and policy cross-entropy provides a robust framework for efficient RL exploration.