Anomaly Detection and Correction of Optimizing Autonomous Systems With Inverse Reinforcement Learning

Area of Science:

• Autonomous Systems
• Control Theory
• Artificial Intelligence

Background:

• Standard condition-based maintenance focuses on non-optimizing systems, failing to address anomalies in systems designed to optimize objectives.
• Autonomous systems may exhibit abnormal behaviors due to misaligned or 'renegade' objective functions, deviating from accepted norms.
• Detecting and correcting such anomalies is crucial for reliable operation of optimizing autonomous agents.

Purpose of the Study:

• To provide a unified framework for anomaly detection and correction in autonomous systems described by differential equations.
• To utilize inverse reinforcement learning (RL) for reconstructing objective functions and intentions of system behaviors.
• To define and address various anomaly types, including objective function and intention anomalies, and associated false alarms.

Main Methods:

• Development of model-free inverse RL algorithms to infer objective functions and system intentions from observed behaviors.
• Implementation of a three-phase procedure: training (offline inference of normal behavior), detection (online inference and comparison), and correction (learning normal behavior).
• Validation using simulations and experimental data from a quadrotor unmanned aerial vehicle (UAV).

Main Results:

• Successfully reconstructed objective functions and intentions for both normal and anomalous system behaviors using inverse RL.
• Demonstrated the framework's ability to accurately detect anomalies and differentiate them from false alarms.
• Showcased the effectiveness of the correction phase in guiding anomalous systems back to normal operational objectives.

Conclusions:

• The proposed inverse RL framework offers a robust method for anomaly detection and correction in optimizing autonomous systems.
• The approach extends beyond traditional fault detection by addressing deviations in system objectives and intentions.
• Experimental validation on a UAV confirms the practical applicability and efficacy of the developed techniques.