Navigating the unknown: Leveraging self-information and diversity in partially observable environments.

Area of Science:

• Artificial Intelligence
• Computational Neuroscience

Background:

• Reinforcement learning (RL) algorithms face challenges in partially observable environments where states can be ambiguous.
• The difficulty of learning in such environments varies, but metrics to quantify this are lacking.

Purpose of the Study:

• Introduce 'dissonance distance' as a metric to estimate learning difficulty in partially observable environments.
• Propose a Spatially Layered Architecture (SLA) inspired by the brain to improve RL robustness against sensory input changes and occlusion.

Main Methods:

• Developed the concept of dissonance distance to quantify environmental learning difficulty.
• Proposed and implemented a Spatially Layered Architecture (SLA) training multiple policies in parallel.
• Evaluated SLA on the partially observable Continuous Mountain Car environment with realistic noise and occlusion.

Main Results:

• Demonstrated that self-information (e.g., internal oscillations, memory) increases dissonance distance, simplifying learning.
• SLA showed learnability and robustness against sensory noise and occlusion.
• SLA adaptively adjusts external information processing for changing environments.

Conclusions:

• Dissonance distance offers a novel way to understand and predict RL learning difficulty.
• The brain-inspired SLA provides an effective, robust solution for RL in dynamic, partially observable environments.
• Multi-policy approaches like SLA may offer insights into complex neural mechanisms, such as dopamine dynamics.