From free energy to expected energy: Improving energy-based value function approximation in reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Free-Energy based Reinforcement Learning (FERL) is effective for high-dimensional spaces but limited to binary or near-binary state inputs.
• FERL approximates value functions using the negative free energy of Restricted Boltzmann Machines (RBMs).
• Previous work showed FERL robustness improves with free energy scaling related to network size.

Purpose of the Study:

• To enhance Restricted Boltzmann Machine (RBM) function approximation in reinforcement learning.
• To introduce a novel method, Expected Energy Reinforcement Learning (EERL), capable of handling continuous state inputs.
• To improve upon the performance and applicability of Free-Energy based Reinforcement Learning (FERL).

Main Methods:

• Proposed approximating the value function with the negative expected energy (EERL) instead of negative free energy.
• Utilized Restricted Boltzmann Machines (RBMs) for function approximation.
• Validated EERL across gridworld tasks, SZ-Tetris, and a robot navigation task with image-based inputs.

Main Results:

• EERL outperformed FERL, standard neural networks, and linear function approximation on high-dimensional gridworld tasks.
• EERL achieved state-of-the-art results in both model-free and model-based learning for stochastic SZ-Tetris.
• EERL significantly surpassed FERL and standard neural network approximation in a robot navigation task using raw, noisy RGB images.

Conclusions:

• Expected Energy Reinforcement Learning (EERL) offers a more robust and versatile approach compared to FERL.
• EERL effectively handles continuous state inputs, expanding the applicability of energy-based models in reinforcement learning.
• The proposed EERL method demonstrates significant performance improvements across diverse and challenging reinforcement learning problems.