Constrained Deep Q-Learning Gradually Approaching Ordinary Q-Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Reinforcement Learning

Background:

• Deep Q Network (DQN) extends Q learning using convolutional neural networks to approximate Q functions for optimal policy derivation.
• DQN utilizes a target network for stable learning, but infrequent updates necessitate large datasets for target value propagation.

Purpose of the Study:

• To introduce Constrained DQN, a novel method that enhances learning efficiency and stability in deep reinforcement learning.
• To address the sample inefficiency of standard DQN by introducing a constraint mechanism for target value updates.

Main Methods:

• Proposed Constrained DQN, which uses the difference between Q function and target network outputs as a constraint on the target value.
• Implemented conservative parameter updates when the difference is large and aggressive updates when the difference is small.
• Observed a decrease in constraint activation as learning progresses, gradually approaching conventional Q learning.

Main Results:

• Constrained DQN converges with a significantly smaller training dataset compared to standard DQN.
• The proposed method demonstrates robustness against variations in target network update frequency and optimizer parameter settings.
• Experimental results indicate Constrained DQN can be effectively integrated with existing approaches like integrated and distributed methods.

Conclusions:

• Constrained DQN offers improved sample efficiency and stability over standard DQN.
• The method is adaptable and can enhance the performance of other advanced deep reinforcement learning techniques.
• Constrained DQN presents a valuable component for developing more efficient and robust reinforcement learning systems.