Qualitative Measurements of Policy Discrepancy for Return-Based Deep Q-Network

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Deep Q-Network (DQN) excels in sequential decision-making.
• Return-based reinforcement learning effectively utilizes sample trajectories.
• Combining these approaches presents a promising research direction.

Purpose of the Study:

• To propose a general framework, R-DQN, integrating DQN and return-based reinforcement learning algorithms.
• To demonstrate performance improvements of DQN through the integration of return-based methods.
• To introduce a strategy for further enhancing R-DQN by measuring policy discrepancy.

Main Methods:

• Developed a unified framework named R-DQN.
• Integrated return-based reinforcement learning algorithms with the DQN.
• Designed a two-measurement strategy to qualitatively assess policy discrepancy.
• Conducted experiments on OpenAI Gym and Atari game tasks.

Main Results:

• The proposed R-DQN framework significantly improves upon traditional DQN performance.
• The policy discrepancy measurement strategy further enhances R-DQN effectiveness.
• Achieved state-of-the-art performance across various experimental tasks.
• Validated the effectiveness of the combined approach and enhancement strategy.

Conclusions:

• The R-DQN framework offers a robust method for combining DQN and return-based reinforcement learning.
• Integrating return-based methods and policy discrepancy analysis boosts reinforcement learning performance.
• The findings demonstrate a significant advancement in reinforcement learning algorithms for complex tasks.