Self-Punishment and Reward Backfill for Deep Q-Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Reinforcement learning (RL) agents learn optimal behaviors by maximizing cumulative rewards from the environment.
• The credit assignment problem arises when rewards are sparse, making it difficult for agents to determine which actions contributed to the outcome.
• Existing RL methods struggle with delayed or infrequent rewards, hindering efficient learning.

Purpose of the Study:

• To propose novel strategies, self-punishment (SP) and reward backfill (RB), inspired by behavioral psychology to address the credit assignment problem in RL.
• To develop methods for agents to intrinsically estimate more informative reward values for actions preceding delayed rewards.
• To ensure proposed strategies maintain policy order and optimality, integrating seamlessly with existing RL algorithms.

Main Methods:

• Introduced self-punishment (SP) to penalize actions leading to undesirable terminal states.
• Implemented reward backfill (RB) to propagate rewards between consecutive rewarded actions.
• Integrated SP and RB into three popular deep reinforcement learning algorithms.
• Evaluated the combined strategies on 30 Atari games.

Main Results:

• The proposed SP and RB strategies were proven to maintain policy order and optimality under specific assumptions, irrespective of the underlying RL algorithm.
• Integration with deep RL approaches demonstrated significant performance improvements across various games.
• Tested methods showed performance enhancements in over 65% of the 30 Atari games.
• Achieved up to a 25-fold performance improvement in certain games after parameter tuning.

Conclusions:

• Self-punishment and reward backfill are effective intrinsic reward estimation strategies for tackling the credit assignment problem in RL.
• These methods enhance the performance of popular deep RL algorithms, particularly in environments with sparse rewards.
• The strategies offer a generalizable approach to improve RL agent learning and policy optimization across diverse applications.