Curiosity-Driven Exploration with Information Bottleneck Representations and Matrix-Based Mutual Information.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Curiosity is a key driver of human exploration and learning.
• Reinforcement learning (RL) often relies on extrinsic rewards, limiting exploration.
• Understanding intrinsic motivation in RL is crucial for developing more autonomous agents.

Purpose of the Study:

• To develop and evaluate a curiosity-based reinforcement learning approach.
• To investigate the impact of intrinsic signals (prediction error, state-action rarity) on exploration efficiency.
• To address the curse of dimensionality in high-dimensional observation spaces using the Information Bottleneck principle.

Main Methods:

• Implemented a curiosity-driven agent using a hybrid intrinsic signal.
• Applied the Information Bottleneck (IB) principle for learning compact, predictive, low-dimensional representations.
• Compared two mutual information formulations: entropy decomposition and matrix-based Rényi entropy.

Main Results:

• The proposed method significantly improved exploration efficiency on the Acrobot task compared to Intrinsic Curiosity Module (ICM), Random Network Distillation (RND), and k-NN novelty.
• Performance on MountainCar indicated that the method is not universally superior in low-dimensional environments.
• IB-shaped representations and matrix-based information objectives showed benefits in high-dimensional or complex dynamic environments.

Conclusions:

• Curiosity-driven exploration can be effectively implemented in reinforcement learning using prediction error and state-action rarity.
• The Information Bottleneck principle is a valuable tool for learning efficient representations in high-dimensional RL.
• The choice of information objective (e.g., matrix-based Rényi entropy) can impact performance, particularly in complex scenarios.