Double XCSF on Target?

The XCS Classifier System (XCS), the most prominent Learning Classifier System (LCS), originally focused on Reinforcement Learning (RL) problems. Over time, emphasis shifted heavily to supervised learning, with some applications in unsupervised learning. Following rekindled interest in LCSs for RL domains, we intend to capitalise on the close relationship between Q-learning and XCS. Except for Experience Replay, hardly any advances built on Q-learning have been investigated in XCS variants such as XCSF. Recognising this, we introduce three extensions inspired by Q-learning derivates: Target prediction inspired by DQN's target networks to improve the learning stability and double target prediction inspired by Double DQN as well as a Double Q-learning mechanism as countermeasures against overestimation. Addressing these two issues, aims to improve the performance of XCSF and the high variance between runs. We apply them to the Maze Problem, Frozen Lake, and Cart Pole. Our observations indicate mixed results: The Double Q-learning mechanism leads to no improvement. Target and double target prediction can lead to observable and also significantly improved performance and can provide variance reduction. This underscores that improving the RL capabilities of XCSF is non-trivial but indicates that adapting Deep Reinforcement Learning mechanisms for XCSF can be advantageous.