Efficient Reinforcement Learning With the Novel N-Step Method and V-Network.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Reinforcement learning (RL) is increasingly used in AI but suffers from low sample efficiency.
• Enhancing sample efficiency is a key research challenge in RL.

Purpose of the Study:

• To address data inefficiency and inaccurate Q-function estimation in RL.
• To propose a novel N-step method combined with V-function regularization.

Main Methods:

• Developed a novel N-step method to extend agent horizon and capture long-term information.
• Introduced a V-function-based regularization technique to mitigate Q-function estimation bias.
• Integrated these methods with classical RL algorithms like DQN, DDPG, and TD3.

Main Results:

• The N-step method reduces estimation variance of the Q-function.
• V-function regularization effectively mitigates Q-function estimation bias.
• Combined methods significantly improve sample efficiency and Q-function accuracy in RL.

Conclusions:

• The proposed N-step method and V-function regularization effectively tackle low sample efficiency and inaccurate Q-function estimation in RL.
• Experiments show consistent performance improvements over classical algorithms in both discrete and continuous action spaces.