Double XCSF on Target?

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• The XCS Classifier System (XCS), a prominent Learning Classifier System (LCS), initially focused on Reinforcement Learning (RL).
• Recent research has explored XCS variants like XCSF, but few Q-learning advancements beyond Experience Replay have been integrated.
• There is a need to enhance XCS's RL capabilities by incorporating modern Q-learning derivatives.

Purpose of the Study:

• To investigate the integration of Deep Reinforcement Learning (DRL) mechanisms into the XCS Classifier System (XCSF) for improved RL performance.
• To address issues of learning stability and overestimation in XCSF by introducing target prediction and Double Q-learning mechanisms.
• To evaluate the effectiveness of these extensions on benchmark RL problems.

Main Methods:

• Implemented three extensions inspired by Q-learning derivatives: DQN-inspired target prediction, Double DQN-inspired double target prediction, and a Double Q-learning mechanism.
• Applied these extensions to the Maze Problem, Frozen Lake, and Cart Pole environments.
• Assessed performance improvements and variance reduction compared to baseline XCSF.

Main Results:

• The Double Q-learning mechanism did not yield significant improvements.
• Target prediction and double target prediction demonstrated observable performance enhancements and variance reduction in XCSF.
• Results indicate that adapting DRL mechanisms can be beneficial for XCSF, though improvements are non-trivial.

Conclusions:

• Integrating target prediction and double target prediction from DRL shows potential for enhancing XCSF performance and stability in RL tasks.
• The Double Q-learning mechanism, however, did not prove effective in this context.
• Further research into DRL adaptations for LCSs like XCSF is warranted to unlock their full RL potential.