A complementary learning systems approach to temporal difference learning.

Area of Science:

• Computational Neuroscience
• Artificial Intelligence
• Machine Learning

Background:

• Complementary Learning Systems (CLS) theory posits two brain systems: slow-learning neocortical and fast-learning hippocampal.
• Deep Reinforcement Learning (RL) models often resemble neocortical slow learning, leading to data inefficiency and inflexibility.
• Integrating both CLS systems may overcome limitations in current deep RL approaches.

Purpose of the Study:

• To propose a novel algorithm, Complementary Temporal Difference Learning (CTDL), inspired by CLS theory.
• To combine Deep Neural Networks (DNNs) with Self-Organizing Maps (SOMs) for enhanced RL.
• To address data inefficiency and flexibility issues in deep RL.

Main Methods:

• Developed CTDL, integrating a DNN with a SOM.
• Utilized Temporal Difference (TD) error to update the SOM.
• Combined SOM and DNN for action value calculation.

Main Results:

• CTDL demonstrated benefits over the Deep Q-Network (DQN) on Grid World, Cart-Pole, and Continuous Mountain Car tasks.
• The algorithm showed utility in action evaluation by leveraging complementary learning systems.
• Results indicate successful extension to both discrete and continuous state and action spaces.

Conclusions:

• Complementary learning systems are valuable for enhancing action evaluation in RL.
• TD error serves as an effective communication signal between learning systems.
• The proposed CTDL approach offers a flexible and data-efficient alternative to traditional deep RL methods.