An immediate-return reinforcement learning for the atypical Markov decision processes.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Atypical Markov decision processes (MDPs) involve maximizing immediate returns in single state transitions, applicable to complex dynamic problems.
• Existing deep reinforcement learning (RL) algorithms are optimized for long-term returns, inefficiently utilizing resources for atypical MDPs.
• Current RL methods suffer from value function estimation errors, leading to suboptimal policies in these specific problem types.

Purpose of the Study:

• To develop an efficient immediate-return algorithm for atypical MDPs with continuous action spaces.
• To address limitations of existing RL algorithms in terms of computational waste and policy performance for atypical MDPs.

Main Methods:

• Proposed an immediate-return algorithm specifically designed for atypical MDPs.
• Introduced an unbiased and low-variance target Q-value estimation.
• Implemented a simplified network framework for enhanced efficiency.

Main Results:

• The proposed algorithm demonstrated superior learning efficiency compared to Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO).
• Achieved a higher effective rate of control in complex dynamic tasks.
• Showcased significant advantages in computing resource utilization.

Conclusions:

• The novel immediate-return algorithm effectively addresses the limitations of traditional RL for atypical MDPs.
• The approach offers a more efficient and effective solution for problems like football trajectory control and parameter identification.
• Validated through simulations of uncertain football passing and chipping scenarios.