Differentiable Logic Policy for Interpretable Deep Reinforcement Learning: A Study From an Optimization Perspective.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Reinforcement Learning

Background:

• Policy interpretability is a significant challenge in Deep Reinforcement Learning (DRL).
• Current DRL methods often lack transparency, hindering understanding and trust.
• Representing policies using symbolic methods could improve interpretability.

Purpose of the Study:

• To explore interpretable Deep Reinforcement Learning (DRL) by representing policies with Differentiable Inductive Logic Programming (DILP).
• To provide a theoretical and empirical analysis of DILP-based policy learning from an optimization viewpoint.
• To introduce a novel optimization approach for DILP-based policies.

Main Methods:

• Representing DRL policies using Differentiable Inductive Logic Programming (DILP).
• Formulating DILP-based policy learning as a constrained policy optimization problem.
• Proposing and analyzing Mirror Descent for Policy Optimization (MDPO) to handle DILP policy constraints.
• Deriving closed-form regret bounds for MDPO with function approximation.
• Investigating the convexity of DILP-based policies.

Main Results:

• Identified DILP-based policy learning as a constrained optimization problem.
• Developed Mirror Descent for Policy Optimization (MDPO) as an effective solution for DILP policy constraints.
• Derived theoretical regret bounds for MDPO, aiding DRL framework design.
• Empirical results validated the effectiveness of MDPO and its on-policy variant compared to mainstream methods.
• Demonstrated the benefits of MDPO through the study of DILP-based policy convexity.

Conclusions:

• Differentiable Inductive Logic Programming (DILP) offers a promising approach for interpretable Deep Reinforcement Learning (DRL).
• Mirror Descent for Policy Optimization (MDPO) provides a theoretically sound and empirically effective method for optimizing constrained DILP-based policies.
• The proposed framework enhances policy interpretability while maintaining competitive performance in DRL tasks.