Quantifying Reinforcement-Learning Agent's Autonomy, Reliance on Memory and Internalisation of the Environment

Area of Science:

• Artificial Intelligence
• Information Theory
• Computational Neuroscience

Background:

• Agent autonomy is intuitively linked to goal and behavior decoupling from environmental control.
• Quantifying autonomy, especially during agent training, remains a challenge in artificial intelligence.

Purpose of the Study:

• To introduce an algorithm for calculating agent autonomy using an information-theoretic framework.
• To investigate how autonomy levels evolve during the training of reinforcement learning agents.
• To utilize Partial Information Decomposition (PID) to monitor autonomy and environment internalization.

Main Methods:

• Developed an algorithm to compute autonomy in a time step limit approaching infinity.
• Applied the Partial Information Decomposition (PID) framework to analyze reinforcement learning agents.
• Conducted experiments in a grid world and a sequence imitation environment.

Main Results:

• Demonstrated that specific PID terms correlate with acquired rewards in reinforcement learning agents.
• Showed that PID quantifies agent reliance on internal memory versus direct observations.
• Observed correlations between PID terms and agent behavior robustness against observational perturbations.

Conclusions:

• The proposed information-theoretic approach effectively quantifies agent autonomy during learning.
• Partial Information Decomposition provides insights into agent internal states and environmental interaction.
• This framework aids in understanding and designing more autonomous and robust artificial agents.