Signal Novelty Detection as an Intrinsic Reward for Robotics

Area of Science:

• Robotics
• Artificial Intelligence
• Machine Learning

Background:

• Reinforcement learning (RL) in advanced robot control often suffers from sparse rewards, leading to slow convergence.
• Intrinsic rewards, particularly those based on state visitation frequency, can offer denser feedback to guide learning.
• Novelty detection is crucial for identifying unexplored states and providing effective intrinsic rewards.

Purpose of the Study:

• To investigate the efficacy of an Autoencoder deep learning neural network for novelty detection in generating intrinsic rewards.
• To enhance the efficiency and accuracy of robot control in simulated environments using autoencoder-based intrinsic rewards.
• To assess the performance of this approach compared to standard extrinsic rewards.

Main Methods:

• Utilized an Autoencoder deep learning neural network for novelty detection to generate intrinsic rewards.
• Processed simultaneous signals from various sensor types.
• Tested the approach on simulated robotic agents in OpenAI Gym environments (Mountain Car, Acrobot, CartPole, LunarLander).

Main Results:

• Achieved more efficient and accurate robot control in three out of four tested OpenAI Gym environments.
• Demonstrated superior performance with intrinsic rewards compared to standard extrinsic rewards.
• Observed only slight degradation in performance on the Lunar Lander task.

Conclusions:

• Autoencoder-based intrinsic rewards significantly improve reinforcement learning for robot control, enhancing adaptability.
• This method offers a promising solution for developing more dependable autonomous robots in challenging operational domains.
• The approach facilitates better adaptation to dynamic environments and unexpected situations, crucial for exploration and disaster response.