Multistate Temporal Difference Target for Model-Free Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Temporal difference (TD) learning is a core reinforcement learning method.
• TD learning uses a TD target for value function estimation.
• Traditional TD targets rely on a single subsequent state.

Purpose of the Study:

• To propose an enhanced multistate TD (MSTD) target for improved value function estimation.
• To develop actor-critic algorithms incorporating the MSTD target.
• To analyze the convergence properties of Q-learning with MSTD.

Main Methods:

• Developed an enhanced multistate TD (MSTD) target utilizing multiple subsequent states.
• Integrated the MSTD target into actor-critic algorithms with replay buffer management.
• Combined MSTD with Deep Deterministic Policy Optimization (DDPG) and Soft Actor-Critic (SAC).
• Analyzed Q-learning convergence with the MSTD target.

Main Results:

• Algorithms using the MSTD target demonstrated improved learning performance over traditional methods.
• Numerical experiments validated the effectiveness of the MSTD approach.
• Convergence analysis of Q-learning with MSTD was performed.

Conclusions:

• The proposed MSTD target enhances value function estimation accuracy in reinforcement learning.
• MSTD-based actor-critic algorithms show superior learning performance.
• The MSTD concept offers a promising direction for advancing TD learning.