Scaled free-energy based reinforcement learning for robust and efficient learning in high-dimensional state spaces

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Standard function approximation struggles with high-dimensional state-action spaces in reinforcement learning.
• Free-energy based reinforcement learning (FERL) offers a potential solution but requires optimization.
• Efficient learning in complex environments remains a key challenge in AI.

Purpose of the Study:

• To propose and evaluate a scaled version of FERL for more robust and efficient learning.
• To investigate the method's feature extraction capabilities for state representation.
• To assess the impact of exploration strategies on learning performance.

Main Methods:

• Developed a scaled FERL approach where the action-value function is approximated by scaled negative free-energy from a Restricted Boltzmann Machine.
• Applied the method to a digit recognition gridworld task using MNIST images.
• Evaluated performance on a robot visual navigation task with a defined state space.
• Compared results against standard FERL and a two-layered feedforward neural network approximation.

Main Results:

• The scaled FERL method demonstrated effective feature extraction for clustering similar and dissimilar digit images.
• The approach showed robustness across different exploration schedules (initial temperature and discount rate).
• Comparative analysis indicated competitive or superior performance against standard FERL and feedforward networks on tested tasks.

Conclusions:

• Scaled FERL provides a robust and efficient framework for reinforcement learning in high-dimensional spaces.
• The method's ability to extract relevant features aids in state representation and task achievement.
• This work advances reinforcement learning techniques for complex perceptual tasks.