Deep Reinforcement Learning With Modulated Hebbian Plus Q-Network Architecture.

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Partially Observable Markov Decision Processes (POMDPs) present challenges for standard reinforcement learning (RL) algorithms, particularly when observations are confounding and rewards are sparse.
• Temporal Difference (TD) error, crucial for many RL algorithms like Deep Q-Network (DQN), becomes unreliable in such scenarios due to difficulties in deriving accurate error signals from observations.

Purpose of the Study:

• To address the limitations of existing RL algorithms in solving confounding POMDPs.
• To introduce a novel bio-inspired neural architecture, the modulated Hebbian plus Q-network architecture (MOHQA), designed to overcome these challenges.

Main Methods:

• The proposed MOHQA architecture integrates a modulated Hebbian network (MOHN) with a Deep Q-Network (DQN).
• The MOHN utilizes bio-inspired neural traces to bridge temporal delays between actions and rewards, compensating for inaccurate TD errors.
• DQN handles low-level feature extraction and control, while MOHN assists in high-level decision-making by associating rewards with past states and actions.

Main Results:

• Simulations demonstrated that MOHQA significantly improved upon standard DQN performance.
• MOHQA outperformed several state-of-the-art RL algorithms, including Advantage Actor-Critic (A2C), Quantile Regression DQN with Long Short-Term Memory (QRDQN + LSTM), REINFORCE, and Aggregated Memory for Reinforcement Learning (AMRL).
• The proposed architecture showed particular efficacy on difficult POMDPs characterized by confounding stimuli and sparse rewards.

Conclusions:

• The MOHQA architecture offers a robust solution for confounding POMDPs, outperforming existing methods.
• Combining Hebbian associative learning with deep reinforcement learning provides synergistic advantages for complex decision-making tasks.
• This bio-inspired approach demonstrates the potential for novel neural architectures to advance reinforcement learning capabilities.