Inverse Reinforcement Q-Learning Through Expert Imitation for Discrete-Time Systems

Area of Science:

• Robotics and Control Systems
• Artificial Intelligence
• Machine Learning

Background:

• Inverse reinforcement learning (RL) involves a learner agent observing an expert agent to infer its cost function.
• Existing methods often require knowledge of the expert agent's dynamics, limiting their applicability.
• Behavior imitation is crucial for developing intelligent agents that can perform complex tasks.

Purpose of the Study:

• To develop a novel inverse RL approach for discrete-time (DT) agents to solve behavior imitation problems.
• To enable a learner agent to determine an expert's cost function solely from observed behavior trajectories.
• To achieve optimal imitation of expert behavior without prior knowledge of agent dynamics.

Main Methods:

• Formulation of a DT behavior imitation problem where the expert's cost function is unknown.
• Development of an inverse RL scheme combining policy iteration, inverse optimal control, and optimal control.
• Introduction of an inverse reinforcement Q-learning algorithm, an extension of RL Q-learning, independent of agent dynamics.

Main Results:

• The proposed inverse RL approach successfully reconstructs the expert's cost function and feedback gain.
• The inverse reinforcement Q-learning algorithm demonstrates stability, convergence, and optimality.
• The study highlights a key property regarding the non-uniqueness of the solution.

Conclusions:

• The developed inverse RL approach effectively solves the behavior imitation problem for DT agents.
• The inverse reinforcement Q-learning algorithm provides a robust method for learning expert behaviors without needing agent dynamics.
• Simulation experiments validate the effectiveness and practical applicability of the proposed method.