Inference-Based Posteriori Parameter Distribution Optimization.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Encouraging agent exploration is crucial yet challenging in reinforcement learning (RL).
• Distributional representations can enhance exploration but may cause instability and inefficiency.
• Existing methods struggle with stable and efficient parameter distribution learning.

Purpose of the Study:

• To propose a novel algorithm, inference-based posteriori parameter distribution optimization (IPPDO), for accelerating and stabilizing parameter distribution learning in RL.
• To design objective functions for both continuous and discrete action tasks based on probability's evidence lower bound.
• To improve data efficiency in deep RL (DRL) through off-policy learning.

Main Methods:

• Developed IPPDO algorithm for parameter distribution optimization using an inference-based approach.
• Designed specific objective functions for continuous and discrete action spaces.
• Employed multiple neural networks with Retrace to mitigate value function overestimation.
• Introduced an activation function on the standard deviation for adaptive weight sampling.
• Utilized off-policy techniques like experience replay for enhanced data efficiency.

Main Results:

• IPPDO demonstrated improved exploration in the action space across continuous and discrete tasks.
• The algorithm achieved higher rewards more rapidly compared to prevailing DRL algorithms.
• Experiments on OpenAI Gym and MuJoCo platforms confirmed IPPDO's algorithmic stability.
• IPPDO effectively balances fixed parameter values and distributional representations.

Conclusions:

• IPPDO offers a stable and efficient method for parameter distribution learning in DRL.
• The algorithm significantly enhances exploration capabilities and learning speed.
• IPPDO presents a promising advancement for off-policy deep reinforcement learning, improving data efficiency and performance.