Enhanced exploration in reinforcement learning using graph neural network based intrinsic reward mechanism

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• The exploration-exploitation dilemma is a key challenge in Reinforcement Learning (RL), impacting agent learning efficiency.
• Existing intrinsic reward strategies often struggle to effectively guide agents toward novel states.

Purpose of the Study:

• To propose a novel framework, Graph Neural Network-based Intrinsic Reward Learning (GNN-IRL), for addressing the exploration-exploitation trade-off in RL.
• To leverage Graph Neural Networks (GNNs) for modeling state transitions and relationships to compute intrinsic rewards.

Main Methods:

• Developed a GNN-IRL framework utilizing GNNs to represent environment states and their relationships.
• Computed intrinsic rewards based on graph centrality and inverse degree analysis to prioritize novel state exploration.
• Discretized continuous state variables to construct state graphs for GNN-IRL implementation.

Main Results:

• GNN-IRL demonstrated superior performance across four benchmark environments (CartPole-v1, MountainCar-v0, Taxi-v3, LunarLander-v3).
• The framework significantly improved convergence rate, cumulative reward, exploration efficiency, and state coverage compared to state-of-the-art methods.
• GNN-IRL effectively balanced exploration and exploitation, enhancing sample efficiency and accelerating policy learning.

Conclusions:

• GNN-IRL offers an effective approach to enhance exploration in discrete Reinforcement Learning domains.
• The GNN-based intrinsic reward mechanism successfully guides agents to explore novel states more efficiently.
• This framework shows promise for improving sample efficiency and learning speed in RL applications.