Map-based experience replay: a memory-efficient solution to catastrophic forgetting in reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Cognitive Science

Background:

• Deep reinforcement learning (RL) agents are prone to catastrophic forgetting, where new training data overwrites previously learned knowledge.
• Traditional replay memories store state transitions without considering redundancy, leading to inefficient memory usage.

Purpose of the Study:

• To develop a novel cognitive-inspired replay memory system for deep reinforcement learning.
• To address the issue of catastrophic forgetting and improve memory efficiency in RL agents.

Main Methods:

• Implemented a replay memory approach based on the Grow-When-Required (GWR) self-organizing network.
• Organized stored state transitions into a network of state nodes and transition edges, merging similar samples.
• Utilized a map-based mental model to structure experience replay.

Main Results:

• The proposed method significantly reduced memory size by creating a concise, environment-model-like network.
• Merging similar samples increased the relevancy and pairwise distance between stored transitions.
• Achieved significant memory reduction with only minor performance decrements.

Conclusions:

• Map-based experience replay offers an effective strategy for memory reduction in deep reinforcement learning.
• The cognitive-inspired GWR network provides a promising direction for mitigating catastrophic forgetting.
• This approach balances memory efficiency with learning performance in RL agents.