Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze | JoVE Visualize

Area of Science:

Artificial Intelligence
Computational Neuroscience
Machine Learning

Background:

Artificial agents often face noisy environments, challenging their learning and adaptation capabilities.
Markov decision processes (MDPs) are foundational for sequential decision-making, but their performance can degrade with environmental uncertainty.
Successor Features (SF) and Predecessor Features (PF) offer advanced methods for learning representations and facilitating transfer learning.

Purpose of the Study:

To quantify the impact of hyperparameter tuning on the adaptability of artificial agents using SF and PF algorithms in a noisy T-maze.
To identify optimal settings for the reward learning rate (αr) and eligibility trace decay rate (λ) to improve agent performance.
To analyze the relationships between adaptation metrics and hyperparameter configurations.

Main Methods:

Agents employing Markov decision processes (MDPs), successor features (SF), and predecessor features (PF) were tested in a noisy T-maze.
Hyperparameter sensitivity analysis was performed by varying the reward learning rate (αr) and eligibility trace decay rate (λ).
Adaptation was evaluated using metrics such as cumulative reward, step length, adaptation rate, and adaptation step length, with Spearman's correlation and linear regression used for analysis.

Main Results:

A reward learning rate (αr) of 0.9 consistently demonstrated superior adaptation across all measured metrics at a noise level of 0.05.
The optimal eligibility trace decay rate (λ) was found to be context-dependent, varying across different adaptation metrics.
Significant correlations were observed between adaptation metrics, highlighting the interconnectedness of performance indicators.

Conclusions:

Hyperparameter optimization is crucial for enhancing the performance and transfer learning capabilities of learning algorithms like SF and PF.
The study provides valuable insights into the effective configuration of learning algorithms for tasks involving environmental uncertainty.
Findings contribute to the development of more robust and adaptive artificial intelligence systems applicable to both AI and neuroscience research.