Robust Inverse Q-Learning for Continuous-Time Linear Systems in Adversarial Environments.

Area of Science:

• Robotics
• Machine Learning
• Control Theory

Background:

• Imitation learning enables agents to learn from expert demonstrations.
• Reconstructing an expert's cost function is crucial for effective imitation.
• Adversarial disturbances pose a challenge in learning accurate expert behaviors.

Purpose of the Study:

• To propose robust inverse Q-learning algorithms for imitation learning.
• To enable a learner to mimic an expert's states and control inputs.
• To reconstruct an unknown expert cost function robustly, independent of system models and disturbances.

Main Methods:

• Developed offline inverse Q-learning with inner Q-learning and outer inverse optimal control loops.
• Extended the offline algorithm to an online version for real-time imitation.
• Utilized four functional approximators in the online method: critic, two actors, and a state-reward neural network.

Main Results:

• The proposed inverse Q-learning algorithms are robust to system model independence and differing cost function parameters/disturbances.
• The online algorithm simultaneously approximates Q-function parameters and learner state-reward.
• Rigorous convergence and stability proofs guarantee algorithm performance.

Conclusions:

• Robust inverse Q-learning algorithms effectively address imitation learning challenges with adversarial disturbances.
• The developed offline and online methods provide a comprehensive solution for mimicking expert behavior.
• The online approach offers real-time adaptation and approximation capabilities.