Transfer of Temporal Logic Formulas in Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Reinforcement learning (RL) benefits from transferring high-level knowledge between tasks.
• Temporal tasks, where event timing is crucial, present unique challenges for knowledge transfer.
• Existing methods often lack specific mechanisms for handling temporal dependencies in knowledge transfer.

Purpose of the Study:

• To develop a transfer learning approach for temporal tasks based on logical similarity.
• To formalize the notion of logical transferability between temporal tasks.
• To improve the sample efficiency of reinforcement learning in target tasks through knowledge transfer.

Main Methods:

• Proposing an inference technique to extract metric interval temporal logic (MITL) formulas from labeled trajectories.
• Constructing timed automata from sequential conjunctive subformulas of inferred MITL formulas.
• Performing RL on an extended state space including timed automata components.
• Transferring extended Q-functions between similar temporal tasks based on established mappings.

Main Results:

• Demonstrating improved sampling efficiency for the target task by up to one order of magnitude through RL in the extended state space.
• Achieving further improvements in sampling efficiency by up to another order of magnitude using transferred extended Q-functions.
• Quantifying the impact of source and target task similarity on the effectiveness of knowledge transfer.

Conclusions:

• The proposed method effectively transfers knowledge between similar temporal tasks, significantly accelerating reinforcement learning.
• Logical transferability, formalized through MITL and timed automata, is a key enabler for efficient knowledge transfer.
• The approach offers a practical solution for enhancing RL performance in time-sensitive applications.